[PATCH v2 0/5] hw/arm/virt: Introduce kvm-steal-time

Andrew Jones posted 5 patches 3 years, 7 months ago
Test docker-quick@centos7 failed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200805091640.11134-1-drjones@redhat.com
Maintainers: Laurent Vivier <lvivier@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>, Christian Borntraeger <borntraeger@de.ibm.com>, David Gibson <david@gibson.dropbear.id.au>, Richard Henderson <rth@twiddle.net>, Eduardo Habkost <ehabkost@redhat.com>, Halil Pasic <pasic@linux.ibm.com>, Thomas Huth <thuth@redhat.com>, Cornelia Huck <cohuck@redhat.com>, Peter Maydell <peter.maydell@linaro.org>, Marcel Apfelbaum <marcel.apfelbaum@gmail.com>, "Michael S. Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com>
There is a newer version of this series
docs/system/arm/cpu-features.rst |  11 +++
hw/arm/virt.c                    | 119 +++++++++++++++++++++++--------
hw/core/machine.c                |   3 +
hw/i386/pc.c                     |   6 +-
hw/i386/pc_piix.c                |  14 +++-
hw/i386/pc_q35.c                 |  13 +++-
hw/ppc/spapr.c                   |  15 +++-
hw/s390x/s390-virtio-ccw.c       |  14 +++-
include/hw/arm/virt.h            |   5 ++
include/hw/boards.h              |   3 +
include/hw/i386/pc.h             |   3 +
target/arm/cpu.c                 |  10 +++
target/arm/cpu.h                 |   4 ++
target/arm/kvm.c                 |  20 ++++++
target/arm/kvm32.c               |   5 ++
target/arm/kvm64.c               |  77 ++++++++++++++++++--
target/arm/kvm_arm.h             |  74 ++++++++++++++-----
target/arm/monitor.c             |   2 +-
tests/qtest/arm-cpu-features.c   |  25 +++++--
19 files changed, 358 insertions(+), 65 deletions(-)
[PATCH v2 0/5] hw/arm/virt: Introduce kvm-steal-time
Posted by Andrew Jones 3 years, 7 months ago
v2:
  - Changed the introduction of the feature to 5.2 from 5.1
    (The 5.2 machine type patch posted by Cornelia was thrown
     in for completeness, but I suppose that'll get picked up
     separately.)
  - Added a patch adding g_assert_not_reached() to many KVM
    stubs. (This isn't exactly related to the series, but the
    series does add two more stubs that can now be added in
    the same way.)
  - Fixed a patch that wasn't suppose to have a functional
    change, but did (fdt_add_gic_node() shouldn't generate
    the node without the PMU) [Peter]
  - Pass sysmem to virt_cpu_post_init() [Peter]
  - Introduced a define for the pvtime struct size
  - Calculate the pvtime memory region size based on max-cpus
    and host-page-size [Beata]
  - Renamed kvm_no_steal_time to no_kvm_steal_time [Peter]
  - Fixed a parameter misordering with object_property_set_bool()
  - Added a comment explaining why the feature isn't supported
    for AArch32 guests
  - Changed a !kvm_steal_time to a kvm_steal_time == ON_OFF_AUTO_OFF
    as it should be
  - Picked up one r-b from Peter


KVM supports the ability to publish the amount of time that VCPUs
were runnable, but not running due to other host threads running
instead, to the guest. The guest scheduler may use that information
when making decisions and the guest may expose it to its userspace
(Linux publishes this information in /proc/stat). This feature is
called "steal time" as it represents the amount of time stolen from
a guest by scheduling out its VCPUs. To enable this feature KVM
userspace must provide a memory region that will be used to publish
the information to the guest. The memory region is typical migratable
region. The GPA of the region is given to KVM through a VCPU device
ioctl interface. This feature is only available for 64-bit hosts
running 64-bit guests.

This series provides the QEMU support of this feature. It will
be enabled by default for 5.2 machine types and later, but may
be disabled with a new CPU property "kvm-steal-time".

While testing migration it was observed that the amount of
steal time as viewed by the guest was getting reset on each
migration. Patch 4/6 of a pvtime fix series posted[*] for KVM
should fix that. Also, we may still want to change the way we
probe KVM for the feature in this QEMU series to the new KVM
cap proposed in that KVM series.

Migration testing:

* virt-5.1 can migrate as usual, no steal-time enabled

* virt-5.2 can migrate between hosts with steal-time enabled
  (the default) and disabled when both hosts support steal-time

* virt-5.2 with steal-time disabled can migrate to a host that
  does not support steal-time

* virt-5.2 with steal-time enabled will cleanly fail when migrating
  to a host that does not support steal-time

* virt-5.2 without the kvm-steal-time property specified can
  boot on a host that does not support the feature - the feature
  will be disabled. However, if the guest is migrated to a host
  that does support the feature, then after the guest has reboot
  the feature will become available with the guest kernel supports
  it (this is the nature of migrating guests that use '-cpu host').
  Additionally, once this guest has been migrated to a host that
  does have the feature, then it will fail (cleanly) to migrate to
  a host that does not have the feature. If this behavior isn't
  desired, then the user should explicitly disable steal-time with
  the kvm-steal-time property if they want to boot a 5.2 guest on
  a host that doesn't support steal-time.

[*] https://lists.cs.columbia.edu/pipermail/kvmarm/2020-August/041823.html

Thanks,
drew


Andrew Jones (4):
  target/arm/kvm: Make uncalled stubs explicitly unreachable
  hw/arm/virt: Move post cpu realize check into its own function
  hw/arm/virt: Move kvm pmu setup to virt_cpu_post_init
  hw/arm/virt: Implement kvm-steal-time

Cornelia Huck (1):
  hw: add compat machines for 5.2

 docs/system/arm/cpu-features.rst |  11 +++
 hw/arm/virt.c                    | 119 +++++++++++++++++++++++--------
 hw/core/machine.c                |   3 +
 hw/i386/pc.c                     |   6 +-
 hw/i386/pc_piix.c                |  14 +++-
 hw/i386/pc_q35.c                 |  13 +++-
 hw/ppc/spapr.c                   |  15 +++-
 hw/s390x/s390-virtio-ccw.c       |  14 +++-
 include/hw/arm/virt.h            |   5 ++
 include/hw/boards.h              |   3 +
 include/hw/i386/pc.h             |   3 +
 target/arm/cpu.c                 |  10 +++
 target/arm/cpu.h                 |   4 ++
 target/arm/kvm.c                 |  20 ++++++
 target/arm/kvm32.c               |   5 ++
 target/arm/kvm64.c               |  77 ++++++++++++++++++--
 target/arm/kvm_arm.h             |  74 ++++++++++++++-----
 target/arm/monitor.c             |   2 +-
 tests/qtest/arm-cpu-features.c   |  25 +++++--
 19 files changed, 358 insertions(+), 65 deletions(-)

-- 
2.25.4


Re: [PATCH v2 0/5] hw/arm/virt: Introduce kvm-steal-time
Posted by Andrew Jones 3 years, 6 months ago
FYI, the new KVM cap is on its way into the kernel and Peter has submit
patches to remove kvm32 from qemu. So I'll be refreshing this series to
incorporate both those changes.

Thanks,
drew


On Wed, Aug 05, 2020 at 11:16:35AM +0200, Andrew Jones wrote:
> v2:
>   - Changed the introduction of the feature to 5.2 from 5.1
>     (The 5.2 machine type patch posted by Cornelia was thrown
>      in for completeness, but I suppose that'll get picked up
>      separately.)
>   - Added a patch adding g_assert_not_reached() to many KVM
>     stubs. (This isn't exactly related to the series, but the
>     series does add two more stubs that can now be added in
>     the same way.)
>   - Fixed a patch that wasn't suppose to have a functional
>     change, but did (fdt_add_gic_node() shouldn't generate
>     the node without the PMU) [Peter]
>   - Pass sysmem to virt_cpu_post_init() [Peter]
>   - Introduced a define for the pvtime struct size
>   - Calculate the pvtime memory region size based on max-cpus
>     and host-page-size [Beata]
>   - Renamed kvm_no_steal_time to no_kvm_steal_time [Peter]
>   - Fixed a parameter misordering with object_property_set_bool()
>   - Added a comment explaining why the feature isn't supported
>     for AArch32 guests
>   - Changed a !kvm_steal_time to a kvm_steal_time == ON_OFF_AUTO_OFF
>     as it should be
>   - Picked up one r-b from Peter
> 
> 
> KVM supports the ability to publish the amount of time that VCPUs
> were runnable, but not running due to other host threads running
> instead, to the guest. The guest scheduler may use that information
> when making decisions and the guest may expose it to its userspace
> (Linux publishes this information in /proc/stat). This feature is
> called "steal time" as it represents the amount of time stolen from
> a guest by scheduling out its VCPUs. To enable this feature KVM
> userspace must provide a memory region that will be used to publish
> the information to the guest. The memory region is typical migratable
> region. The GPA of the region is given to KVM through a VCPU device
> ioctl interface. This feature is only available for 64-bit hosts
> running 64-bit guests.
> 
> This series provides the QEMU support of this feature. It will
> be enabled by default for 5.2 machine types and later, but may
> be disabled with a new CPU property "kvm-steal-time".
> 
> While testing migration it was observed that the amount of
> steal time as viewed by the guest was getting reset on each
> migration. Patch 4/6 of a pvtime fix series posted[*] for KVM
> should fix that. Also, we may still want to change the way we
> probe KVM for the feature in this QEMU series to the new KVM
> cap proposed in that KVM series.
> 
> Migration testing:
> 
> * virt-5.1 can migrate as usual, no steal-time enabled
> 
> * virt-5.2 can migrate between hosts with steal-time enabled
>   (the default) and disabled when both hosts support steal-time
> 
> * virt-5.2 with steal-time disabled can migrate to a host that
>   does not support steal-time
> 
> * virt-5.2 with steal-time enabled will cleanly fail when migrating
>   to a host that does not support steal-time
> 
> * virt-5.2 without the kvm-steal-time property specified can
>   boot on a host that does not support the feature - the feature
>   will be disabled. However, if the guest is migrated to a host
>   that does support the feature, then after the guest has reboot
>   the feature will become available with the guest kernel supports
>   it (this is the nature of migrating guests that use '-cpu host').
>   Additionally, once this guest has been migrated to a host that
>   does have the feature, then it will fail (cleanly) to migrate to
>   a host that does not have the feature. If this behavior isn't
>   desired, then the user should explicitly disable steal-time with
>   the kvm-steal-time property if they want to boot a 5.2 guest on
>   a host that doesn't support steal-time.
> 
> [*] https://lists.cs.columbia.edu/pipermail/kvmarm/2020-August/041823.html
> 
> Thanks,
> drew
> 
> 
> Andrew Jones (4):
>   target/arm/kvm: Make uncalled stubs explicitly unreachable
>   hw/arm/virt: Move post cpu realize check into its own function
>   hw/arm/virt: Move kvm pmu setup to virt_cpu_post_init
>   hw/arm/virt: Implement kvm-steal-time
> 
> Cornelia Huck (1):
>   hw: add compat machines for 5.2
> 
>  docs/system/arm/cpu-features.rst |  11 +++
>  hw/arm/virt.c                    | 119 +++++++++++++++++++++++--------
>  hw/core/machine.c                |   3 +
>  hw/i386/pc.c                     |   6 +-
>  hw/i386/pc_piix.c                |  14 +++-
>  hw/i386/pc_q35.c                 |  13 +++-
>  hw/ppc/spapr.c                   |  15 +++-
>  hw/s390x/s390-virtio-ccw.c       |  14 +++-
>  include/hw/arm/virt.h            |   5 ++
>  include/hw/boards.h              |   3 +
>  include/hw/i386/pc.h             |   3 +
>  target/arm/cpu.c                 |  10 +++
>  target/arm/cpu.h                 |   4 ++
>  target/arm/kvm.c                 |  20 ++++++
>  target/arm/kvm32.c               |   5 ++
>  target/arm/kvm64.c               |  77 ++++++++++++++++++--
>  target/arm/kvm_arm.h             |  74 ++++++++++++++-----
>  target/arm/monitor.c             |   2 +-
>  tests/qtest/arm-cpu-features.c   |  25 +++++--
>  19 files changed, 358 insertions(+), 65 deletions(-)
> 
> -- 
> 2.25.4
>