Synchronizing the list of cpu features and models with qemu is a recurring
task in libvirt. For x86, this is done by reading qom-list-properties for
max-x86_64-cpu and manually filtering out everthing that does not look like
a feature name, as well as parsing target/i386/cpu.c for cpu models.

This is a flawed, tedious and error-prone procedure. Ideally, qemu
and libvirt would query a common source for cpu feature and model
related information. Meanwhile, converting this information into an easier
to parse format would help libvirt a lot.

This patch series converts the cpu feature information present in
target/i386/cpu.c (`feature_word_info`) into a yaml file and adds a
script to generate the c code from this data.

A patch set to convert the cpu model data (`builtin_x86_defs`) in the
same way will follow.

v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-08/msg02005.html

Changes since v1:
* Incorporated changes from
  https://lists.nongnu.org/archive/html/qemu-devel/2023-08/msg04241.html.
* Changed data format from xml to yaml, as proposed in
  https://lists.nongnu.org/archive/html/qemu-devel/2023-09/msg01033.html.
  Using json has some drawbacks, see
  https://lists.nongnu.org/archive/html/qemu-devel/2023-08/msg03384.html.
* Rebased on top of current master. Features added in the meantime:
  amx-complex and vmx-enable-user-wait-pause
* Split up the reformatting of feature_word_info.c.inc to make it easier
  to review. These patches could be squashed together before merging.

Tim Wiederhake (10):
  target/i386: Add missing feature names in FEAT_VMX_EPT_VPID_CAPS
  target/i386: Fix feature names in FEAT_VMX_EPT_VPID_CAPS
  target/i386: Fix duplicated feature name in FEAT_KVM
  target/i386: Split out feature_word_info
  target/i386: Translate feature_word_info to yaml
  target/i386: Format feature_word_info.c.inc: Remove comments
  target/i386: Format feature_word_info.c.inc: feat_names
  target/i386: Format feature_word_info.c.inc: Unfold cpuid member
  target/i386: Format feature_word_info.c.inc: Whitespaces and trailing
    commas
  target/i386: Autogenerate feature_word_info.c.inc

 target/i386/cpu.c                   |  677 +-----------------
 target/i386/feature_word_info.c.inc | 1030 +++++++++++++++++++++++++++
 target/i386/feature_word_info.py    |   62 ++
 target/i386/feature_word_info.yaml  |  697 ++++++++++++++++++
 4 files changed, 1790 insertions(+), 676 deletions(-)
 create mode 100644 target/i386/feature_word_info.c.inc
 create mode 100755 target/i386/feature_word_info.py
 create mode 100644 target/i386/feature_word_info.yaml

-- 
2.39.2

On Fri,  8 Sep 2023 14:45:24 +0200
Tim Wiederhake <twiederh@redhat.com> wrote:

> Synchronizing the list of cpu features and models with qemu is a recurring
> task in libvirt. For x86, this is done by reading qom-list-properties for
> max-x86_64-cpu and manually filtering out everthing that does not look like
> a feature name, as well as parsing target/i386/cpu.c for cpu models.

modulo fixing typos/name conflicts in 1st 3 patches,

I don't think it's a great idea for libvirt (or any other user) to parse
QEMU source (whether it's C code or yaml) or other way around for users
to influence QEMU internals.

QEMU does provides QMP interface for introspection of CPU models and it
should be used as a mean to discover new CPU models as well as new
feature names. If something is missing there we should fix it up
to make it usable for libvirt.

> This is a flawed, tedious and error-prone procedure. Ideally, qemu
> and libvirt would query a common source for cpu feature and model
> related information. Meanwhile, converting this information into an easier
> to parse format would help libvirt a lot.
> 
> This patch series converts the cpu feature information present in
> target/i386/cpu.c (`feature_word_info`) into a yaml file and adds a
> script to generate the c code from this data.

 
> A patch set to convert the cpu model data (`builtin_x86_defs`) in the
> same way will follow.

while theoretically you can parse feature_word_info with some sort of reliability,
you can't do that reliably with CPU model definitions in builtin_x86_defs
as the later is just a skeleton. Actual CPU models definitions are
a function of used machine type, accelerator and host kernel. 

> 
> v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-08/msg02005.html
> 
> Changes since v1:
> * Incorporated changes from
>   https://lists.nongnu.org/archive/html/qemu-devel/2023-08/msg04241.html.
> * Changed data format from xml to yaml, as proposed in
>   https://lists.nongnu.org/archive/html/qemu-devel/2023-09/msg01033.html.
>   Using json has some drawbacks, see
>   https://lists.nongnu.org/archive/html/qemu-devel/2023-08/msg03384.html.
> * Rebased on top of current master. Features added in the meantime:
>   amx-complex and vmx-enable-user-wait-pause
> * Split up the reformatting of feature_word_info.c.inc to make it easier
>   to review. These patches could be squashed together before merging.
> 
> Tim Wiederhake (10):
>   target/i386: Add missing feature names in FEAT_VMX_EPT_VPID_CAPS
>   target/i386: Fix feature names in FEAT_VMX_EPT_VPID_CAPS
>   target/i386: Fix duplicated feature name in FEAT_KVM
>   target/i386: Split out feature_word_info
>   target/i386: Translate feature_word_info to yaml
>   target/i386: Format feature_word_info.c.inc: Remove comments
>   target/i386: Format feature_word_info.c.inc: feat_names
>   target/i386: Format feature_word_info.c.inc: Unfold cpuid member
>   target/i386: Format feature_word_info.c.inc: Whitespaces and trailing
>     commas
>   target/i386: Autogenerate feature_word_info.c.inc
> 
>  target/i386/cpu.c                   |  677 +-----------------
>  target/i386/feature_word_info.c.inc | 1030 +++++++++++++++++++++++++++
>  target/i386/feature_word_info.py    |   62 ++
>  target/i386/feature_word_info.yaml  |  697 ++++++++++++++++++
>  4 files changed, 1790 insertions(+), 676 deletions(-)
>  create mode 100644 target/i386/feature_word_info.c.inc
>  create mode 100755 target/i386/feature_word_info.py
>  create mode 100644 target/i386/feature_word_info.yaml
>

On Fri, Sep 08, 2023 at 04:48:46PM +0200, Igor Mammedov wrote:
> On Fri,  8 Sep 2023 14:45:24 +0200
> Tim Wiederhake <twiederh@redhat.com> wrote:
> 
> > Synchronizing the list of cpu features and models with qemu is a recurring
> > task in libvirt. For x86, this is done by reading qom-list-properties for
> > max-x86_64-cpu and manually filtering out everthing that does not look like
> > a feature name, as well as parsing target/i386/cpu.c for cpu models.
> 
> modulo fixing typos/name conflicts in 1st 3 patches,
> 
> I don't think it's a great idea for libvirt (or any other user) to parse
> QEMU source (whether it's C code or yaml) or other way around for users
> to influence QEMU internals.

NB It isn't for libvirt to parse at runtime, rather it is for libvirt
maintainers to consume during dev, so libvirt keeps in sync with QEMU
features.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Fri, 8 Sep 2023 17:55:12 +0100
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Fri, Sep 08, 2023 at 04:48:46PM +0200, Igor Mammedov wrote:
> > On Fri,  8 Sep 2023 14:45:24 +0200
> > Tim Wiederhake <twiederh@redhat.com> wrote:
> >   
> > > Synchronizing the list of cpu features and models with qemu is a recurring
> > > task in libvirt. For x86, this is done by reading qom-list-properties for
> > > max-x86_64-cpu and manually filtering out everthing that does not look like
> > > a feature name, as well as parsing target/i386/cpu.c for cpu models.  
> > 
> > modulo fixing typos/name conflicts in 1st 3 patches,
> > 
> > I don't think it's a great idea for libvirt (or any other user) to parse
> > QEMU source (whether it's C code or yaml) or other way around for users
> > to influence QEMU internals.  
> 
> NB It isn't for libvirt to parse at runtime, rather it is for libvirt
> maintainers to consume during dev, so libvirt keeps in sync with QEMU
> features.

As QEMU dev, I'm not fond of code generators as sometimes they make
difficult for me to read, and on to of that inventing new 'language'
to describe features that works on some cases only (not everything
is described in feature array, and for non x86 properties mostly
coded in initfn/realizefn).
(I'd dislike it less if it were part of QMP schema as it gets us
closer to processing '-device' with QMP machinery).

why not use existing QMP interface there as well (or alter it if
it's not sufficient)?

> 
> 
> With regards,
> Daniel

On Mon, 2023-09-11 at 13:26 +0200, Igor Mammedov wrote:
> On Fri, 8 Sep 2023 17:55:12 +0100
> Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > On Fri, Sep 08, 2023 at 04:48:46PM +0200, Igor Mammedov wrote:
> > > On Fri,  8 Sep 2023 14:45:24 +0200
> > > Tim Wiederhake <twiederh@redhat.com> wrote:
> > >   
> > > > Synchronizing the list of cpu features and models with qemu is
> > > > a recurring
> > > > task in libvirt. For x86, this is done by reading qom-list-
> > > > properties for
> > > > max-x86_64-cpu and manually filtering out everthing that does
> > > > not look like
> > > > a feature name, as well as parsing target/i386/cpu.c for cpu
> > > > models.  
> > > 
> > > modulo fixing typos/name conflicts in 1st 3 patches,
> > > 
> > > I don't think it's a great idea for libvirt (or any other user)
> > > to parse
> > > QEMU source (whether it's C code or yaml) or other way around for
> > > users
> > > to influence QEMU internals.  
> > 
> > NB It isn't for libvirt to parse at runtime, rather it is for
> > libvirt
> > maintainers to consume during dev, so libvirt keeps in sync with
> > QEMU
> > features.
> 
> As QEMU dev, I'm not fond of code generators as sometimes they make
> difficult for me to read, and on to of that inventing new 'language'
> to describe features that works on some cases only (not everything
> is described in feature array, and for non x86 properties mostly
> coded in initfn/realizefn).
> (I'd dislike it less if it were part of QMP schema as it gets us
> closer to processing '-device' with QMP machinery).
> 
> why not use existing QMP interface there as well (or alter it if
> it's not sufficient)?
> 
> 
I understand your concern regarding the legibility of generated code.
If you have a look at patches 6 - 9, you will see that the only changes
introduced are white space changes, harmonizing the usage of trailing
commas, breaking lines, and filling the "feat_names" field fully. That,
if anything, makes the code more readable in my opinion.

The format of the yaml file is chosen to mimic struct FeatureWordInfo
as closely as possible, it is effectively a 1:1 mapping.

Regarding using the QMP interface: That is what libvirt is currently
doing, see [1] if you are interested. The problem with that approach is
that the response to

    {
        "execute": "qom-list-properties",
        "arguments": { "typename": "max-x86_64-cpu"}
    }

does not distinguish between cpu features and other information about
that cpu model, thus forcing libvirt to maintain a list of non-
features. Examples of such items are "check", "enforce", "start-
powered-off", or "migratable".

Additionally, the response does not include information about the cpuid
leaf, register and bit, or msr register respectively. This has to be
supplemented manually.

If there is a way to query for this information that I am not aware of,
please let me know.

Ultimately I would love to see libvirt stop querying qemu for x86 cpu
feature information but instead have both qemu and libvirt query a
common source / database for this kind of information, for all
architectures, not only x86.

Regards,
Tim

[1]
https://gitlab.com/libvirt/libvirt/-/blob/master/src/cpu_map/sync_qemu_features_i386.py