Changes since v3:
- Make 'hv-default' override 'hv-*' options which were already set 
  (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
  behave the same way.
- Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
  enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
- Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
  to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
- Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
  support the above mentioned changes.
- Expand qtest to check the above mentioned improvements.

Original description:

Upper layer tools like libvirt want to figure out which Hyper-V features are
supported by the underlying stack (QEMU/KVM) but currently they are unable to
do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
no effect on e.g. QMP's 

query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}

command as we parse Hyper-V features after creating KVM vCPUs and not at
feature expansion time. To support the use-case we first need to make 
KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
vCPU version can't be used that early. This is what KVM part does. With
that done, we can make early Hyper-V feature expansion (this series).

In addition, provide a simple 'hv-default' option which enables (and
requires from KVM) all currently supported Hyper-V enlightenments.
Unlike 'hv-passthrough' mode, this is going to be migratable.

Vitaly Kuznetsov (21):
  i386: keep hyperv_vendor string up-to-date
  i386: invert hyperv_spinlock_attempts setting logic with
    hv_passthrough
  i386: always fill Hyper-V CPUID feature leaves from X86CPU data
  i386: stop using env->features[] for filling Hyper-V CPUIDs
  i386: introduce hyperv_feature_supported()
  i386: introduce hv_cpuid_get_host()
  i386: drop FEAT_HYPERV feature leaves
  i386: introduce hv_cpuid_cache
  i386: split hyperv_handle_properties() into
    hyperv_expand_features()/hyperv_fill_cpuids()
  i386: move eVMCS enablement to hyperv_init_vcpu()
  i386: switch hyperv_expand_features() to using error_setg()
  i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size
  i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one
  i386: use global kvm_state in hyperv_enabled() check
  i386: expand Hyper-V features during CPU feature expansion time
  i386: track explicit 'hv-*' features enablement/disablement
  i386: support 'hv-passthrough,hv-feature=off' on the command line
  i386: be more picky about implicit 'hv-evmcs' enablement
  i386: introduce kvm_hv_evmcs_available()
  i386: provide simple 'hv-default=on' option
  qtest/hyperv: Introduce a simple hyper-v test

 MAINTAINERS                |   1 +
 docs/hyperv.txt            |  16 +-
 target/i386/cpu.c          | 430 ++++++++++++++++++++---------
 target/i386/cpu.h          |  11 +-
 target/i386/kvm/kvm-stub.c |  10 +
 target/i386/kvm/kvm.c      | 535 +++++++++++++++++++++----------------
 target/i386/kvm/kvm_i386.h |   2 +
 tests/qtest/hyperv-test.c  | 312 +++++++++++++++++++++
 tests/qtest/meson.build    |   3 +-
 9 files changed, 950 insertions(+), 370 deletions(-)
 create mode 100644 tests/qtest/hyperv-test.c

-- 
2.29.2

On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote:
> Changes since v3:
> - Make 'hv-default' override 'hv-*' options which were already set 
>   (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
>   behave the same way.
> - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
>   enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
> - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
>   to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
> - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
>   support the above mentioned changes.
> - Expand qtest to check the above mentioned improvements.
> 
> Original description:
> 
> Upper layer tools like libvirt want to figure out which Hyper-V features are
> supported by the underlying stack (QEMU/KVM) but currently they are unable to
> do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
> no effect on e.g. QMP's 
> 
> query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}
> 
> command as we parse Hyper-V features after creating KVM vCPUs and not at
> feature expansion time. To support the use-case we first need to make 
> KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
> vCPU version can't be used that early. This is what KVM part does. With
> that done, we can make early Hyper-V feature expansion (this series).
> 
> In addition, provide a simple 'hv-default' option which enables (and
> requires from KVM) all currently supported Hyper-V enlightenments.
> Unlike 'hv-passthrough' mode, this is going to be migratable.

How is it going to be migratable if the semantics vary depending on
the host kernel KVM reporting features, because different kernels
will expose different features ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote:
> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote:
> > Changes since v3:
> > - Make 'hv-default' override 'hv-*' options which were already set 
> >   (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
> >   behave the same way.
> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
> >   enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
> >   to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
> >   support the above mentioned changes.
> > - Expand qtest to check the above mentioned improvements.
> > 
> > Original description:
> > 
> > Upper layer tools like libvirt want to figure out which Hyper-V features are
> > supported by the underlying stack (QEMU/KVM) but currently they are unable to
> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
> > no effect on e.g. QMP's 
> > 
> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}
> > 
> > command as we parse Hyper-V features after creating KVM vCPUs and not at
> > feature expansion time. To support the use-case we first need to make 
> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
> > vCPU version can't be used that early. This is what KVM part does. With
> > that done, we can make early Hyper-V feature expansion (this series).
> > 
> > In addition, provide a simple 'hv-default' option which enables (and
> > requires from KVM) all currently supported Hyper-V enlightenments.
> > Unlike 'hv-passthrough' mode, this is going to be migratable.
> 
> How is it going to be migratable if the semantics vary depending on
> the host kernel KVM reporting features, because different kernels
> will expose different features ?

"all currently supported" in this context means "all features
supported when the machine type was added", not "all features
supported by the host kernel".

-- 
Eduardo

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote:
>> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote:
>> > Changes since v3:
>> > - Make 'hv-default' override 'hv-*' options which were already set 
>> >   (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
>> >   behave the same way.
>> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
>> >   enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
>> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
>> >   to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
>> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
>> >   support the above mentioned changes.
>> > - Expand qtest to check the above mentioned improvements.
>> > 
>> > Original description:
>> > 
>> > Upper layer tools like libvirt want to figure out which Hyper-V features are
>> > supported by the underlying stack (QEMU/KVM) but currently they are unable to
>> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
>> > no effect on e.g. QMP's 
>> > 
>> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}
>> > 
>> > command as we parse Hyper-V features after creating KVM vCPUs and not at
>> > feature expansion time. To support the use-case we first need to make 
>> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
>> > vCPU version can't be used that early. This is what KVM part does. With
>> > that done, we can make early Hyper-V feature expansion (this series).
>> > 
>> > In addition, provide a simple 'hv-default' option which enables (and
>> > requires from KVM) all currently supported Hyper-V enlightenments.
>> > Unlike 'hv-passthrough' mode, this is going to be migratable.
>> 
>> How is it going to be migratable if the semantics vary depending on
>> the host kernel KVM reporting features, because different kernels
>> will expose different features ?
>
> "all currently supported" in this context means "all features
> supported when the machine type was added", not "all features
> supported by the host kernel".

Yes, exactly.

'hv-passthrough' enables 'everything supported by the host' and this is
not migratable.

'hv-default' requires a certain set of features (depending on the
machine type) so the VM won't start if the host lacks something.

-- 
Vitaly

On Thu, Feb 11, 2021 at 09:30:53AM +0100, Vitaly Kuznetsov wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote:
> >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote:
> >> > Changes since v3:
> >> > - Make 'hv-default' override 'hv-*' options which were already set 
> >> >   (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
> >> >   behave the same way.
> >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
> >> >   enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
> >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
> >> >   to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
> >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
> >> >   support the above mentioned changes.
> >> > - Expand qtest to check the above mentioned improvements.
> >> > 
> >> > Original description:
> >> > 
> >> > Upper layer tools like libvirt want to figure out which Hyper-V features are
> >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to
> >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
> >> > no effect on e.g. QMP's 
> >> > 
> >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}
> >> > 
> >> > command as we parse Hyper-V features after creating KVM vCPUs and not at
> >> > feature expansion time. To support the use-case we first need to make 
> >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
> >> > vCPU version can't be used that early. This is what KVM part does. With
> >> > that done, we can make early Hyper-V feature expansion (this series).
> >> > 
> >> > In addition, provide a simple 'hv-default' option which enables (and
> >> > requires from KVM) all currently supported Hyper-V enlightenments.
> >> > Unlike 'hv-passthrough' mode, this is going to be migratable.
> >> 
> >> How is it going to be migratable if the semantics vary depending on
> >> the host kernel KVM reporting features, because different kernels
> >> will expose different features ?
> >
> > "all currently supported" in this context means "all features
> > supported when the machine type was added", not "all features
> > supported by the host kernel".
> 
> Yes, exactly.
> 
> 'hv-passthrough' enables 'everything supported by the host' and this is
> not migratable.
> 
> 'hv-default' requires a certain set of features (depending on the
> machine type) so the VM won't start if the host lacks something.

Ok, so I presume HV features will only be added to hv-default when we
know they are available in the oldest kernel we are targetting ? Upsteam
is more conservative in this respect than downstreams,  the latter can
guarantee much more modern kernels.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Feb 11, 2021 at 09:30:53AM +0100, Vitaly Kuznetsov wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>> 
>> > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote:
>> >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote:
>> >> > Changes since v3:
>> >> > - Make 'hv-default' override 'hv-*' options which were already set 
>> >> >   (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
>> >> >   behave the same way.
>> >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
>> >> >   enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
>> >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
>> >> >   to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
>> >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
>> >> >   support the above mentioned changes.
>> >> > - Expand qtest to check the above mentioned improvements.
>> >> > 
>> >> > Original description:
>> >> > 
>> >> > Upper layer tools like libvirt want to figure out which Hyper-V features are
>> >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to
>> >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
>> >> > no effect on e.g. QMP's 
>> >> > 
>> >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}
>> >> > 
>> >> > command as we parse Hyper-V features after creating KVM vCPUs and not at
>> >> > feature expansion time. To support the use-case we first need to make 
>> >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
>> >> > vCPU version can't be used that early. This is what KVM part does. With
>> >> > that done, we can make early Hyper-V feature expansion (this series).
>> >> > 
>> >> > In addition, provide a simple 'hv-default' option which enables (and
>> >> > requires from KVM) all currently supported Hyper-V enlightenments.
>> >> > Unlike 'hv-passthrough' mode, this is going to be migratable.
>> >> 
>> >> How is it going to be migratable if the semantics vary depending on
>> >> the host kernel KVM reporting features, because different kernels
>> >> will expose different features ?
>> >
>> > "all currently supported" in this context means "all features
>> > supported when the machine type was added", not "all features
>> > supported by the host kernel".
>> 
>> Yes, exactly.
>> 
>> 'hv-passthrough' enables 'everything supported by the host' and this is
>> not migratable.
>> 
>> 'hv-default' requires a certain set of features (depending on the
>> machine type) so the VM won't start if the host lacks something.
>
> Ok, so I presume HV features will only be added to hv-default when we
> know they are available in the oldest kernel we are targetting ? Upsteam
> is more conservative in this respect than downstreams,  the latter can
> guarantee much more modern kernels.
>

Yes, it is kind of an open question when a feature gets 'promoted' to
'hv-default'. Currently, the latest feature we include is
'HYPERV_FEAT_STIMER_DIRECT' which dates back to Linux 5.0. It is also
possible to use something like

'hv-default,hv-stimer-direct=off,...'

when running on an older kernel (and this is still migratable).

-- 
Vitaly

On Thu, Feb 11, 2021 at 10:34:15AM +0100, Vitaly Kuznetsov wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Thu, Feb 11, 2021 at 09:30:53AM +0100, Vitaly Kuznetsov wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> >> 
> >> > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote:
> >> >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote:
> >> >> > Changes since v3:
> >> >> > - Make 'hv-default' override 'hv-*' options which were already set 
> >> >> >   (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough'
> >> >> >   behave the same way.
> >> >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid
> >> >> >   enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX.
> >> >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch
> >> >> >   to make 'hv-passthrough' semantics match the newly introduced 'hv-default'.
> >> >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to
> >> >> >   support the above mentioned changes.
> >> >> > - Expand qtest to check the above mentioned improvements.
> >> >> > 
> >> >> > Original description:
> >> >> > 
> >> >> > Upper layer tools like libvirt want to figure out which Hyper-V features are
> >> >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to
> >> >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
> >> >> > no effect on e.g. QMP's 
> >> >> > 
> >> >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}
> >> >> > 
> >> >> > command as we parse Hyper-V features after creating KVM vCPUs and not at
> >> >> > feature expansion time. To support the use-case we first need to make 
> >> >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
> >> >> > vCPU version can't be used that early. This is what KVM part does. With
> >> >> > that done, we can make early Hyper-V feature expansion (this series).
> >> >> > 
> >> >> > In addition, provide a simple 'hv-default' option which enables (and
> >> >> > requires from KVM) all currently supported Hyper-V enlightenments.
> >> >> > Unlike 'hv-passthrough' mode, this is going to be migratable.
> >> >> 
> >> >> How is it going to be migratable if the semantics vary depending on
> >> >> the host kernel KVM reporting features, because different kernels
> >> >> will expose different features ?
> >> >
> >> > "all currently supported" in this context means "all features
> >> > supported when the machine type was added", not "all features
> >> > supported by the host kernel".
> >> 
> >> Yes, exactly.
> >> 
> >> 'hv-passthrough' enables 'everything supported by the host' and this is
> >> not migratable.
> >> 
> >> 'hv-default' requires a certain set of features (depending on the
> >> machine type) so the VM won't start if the host lacks something.
> >
> > Ok, so I presume HV features will only be added to hv-default when we
> > know they are available in the oldest kernel we are targetting ? Upsteam
> > is more conservative in this respect than downstreams,  the latter can
> > guarantee much more modern kernels.
> >
> 
> Yes, it is kind of an open question when a feature gets 'promoted' to
> 'hv-default'. Currently, the latest feature we include is
> 'HYPERV_FEAT_STIMER_DIRECT' which dates back to Linux 5.0. It is also
> possible to use something like

Upstream we have a clear set of targetted OS platforms:

  https://qemu.readthedocs.io/en/latest/system/build-platforms.html

This will inform what our minimum possible kernel version will be,
which should influence when we can promote something.

Downstream's of course have their own min kernel model, so they can
do things differently if they have their own machine types.

> 'hv-default,hv-stimer-direct=off,...'
> 
> when running on an older kernel (and this is still migratable).

At that point the mgmt app needs to know exactly what features the
host supports, and if they know that, there's no real need to use
hv-default in the first place. IOW, I think we need to strive to
ensure "hv-default" is always usable on supported platforms without
needing to know about turning off things.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|