From nobody Mon Sep 16 18:52:56 2024 Delivered-To: importer@patchew.org Received-SPF: none (zohomail.com: 8.43.85.245 is neither permitted nor denied by domain of lists.libvirt.org) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; spf=none (zohomail.com: 8.43.85.245 is neither permitted nor denied by domain of lists.libvirt.org) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 170688732068116.81200865437677; Fri, 2 Feb 2024 07:22:00 -0800 (PST) Received: by lists.libvirt.org (Postfix, from userid 996) id 604AC1977; Fri, 2 Feb 2024 10:21:59 -0500 (EST) Received: from lists.libvirt.org.85.43.8.in-addr.arpa (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id 937B51B44; Fri, 2 Feb 2024 10:09:40 -0500 (EST) Received: by lists.libvirt.org (Postfix, from userid 996) id E3E751B2A; Fri, 2 Feb 2024 10:09:11 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 66A2A1B1D for ; Fri, 2 Feb 2024 10:09:08 -0500 (EST) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-569-7ZIznC8LOlyK6tEiiSc5DA-1; Fri, 02 Feb 2024 10:09:06 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 483D2837227 for ; Fri, 2 Feb 2024 15:09:06 +0000 (UTC) Received: from work.fritz.box (unknown [10.39.194.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id D2AAC111D792; Fri, 2 Feb 2024 15:09:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.4 X-MC-Unique: 7ZIznC8LOlyK6tEiiSc5DA-1 From: Tim Wiederhake To: devel@lists.libvirt.org Subject: [PATCH 5/5] cpu_map: Rewrite feature sync script Date: Fri, 2 Feb 2024 16:08:57 +0100 Message-ID: <20240202150857.157070-6-twiederh@redhat.com> In-Reply-To: <20240202150857.157070-1-twiederh@redhat.com> References: <20240202150857.157070-1-twiederh@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Message-ID-Hash: HGWUOTPUFKQYFINGTBX5ZPAWEDVCDTUO X-Message-ID-Hash: HGWUOTPUFKQYFINGTBX5ZPAWEDVCDTUO X-MailFrom: twiederh@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Tim Wiederhake X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="utf-8"; x-default="true" Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1706887321360100001 Previously, the script would only detect differences between libvirt's and qemu's list of x86 features, adding those features to libvirt was a manual and error prone procedure. Replace with a script that can generate libvirt's feature list directly from qemu source code. --- src/cpu_map/sync_qemu_features_i386.py | 548 +++++++++++++++++-------- src/cpu_map/x86_features.xml | 7 +- 2 files changed, 382 insertions(+), 173 deletions(-) diff --git a/src/cpu_map/sync_qemu_features_i386.py b/src/cpu_map/sync_qemu= _features_i386.py index dcd58cd590..1fc7991a66 100755 --- a/src/cpu_map/sync_qemu_features_i386.py +++ b/src/cpu_map/sync_qemu_features_i386.py @@ -1,190 +1,400 @@ #!/usr/bin/env python3 =20 import argparse -import json -import subprocess -import xml.etree.ElementTree - - -def ignore_feature(feature): - ignored_features =3D [ - # VIA/Cyrix/Centaur-defined CPU features - # CPUID level 0xC0000001, word 5 - "ace2", - "ace2-en", - "phe", - "phe-en", - "pmm", - "pmm-en", - "xcrypt", - "xcrypt-en", - "xstore", - "xstore-en", - - # non-features - "check", - "cpuid-0xb", - "enforce", - "fill-mtrr-mask", - "full-cpuid-auto-level", - "full-width-write", - "host-cache-info", - "host-phys-bits", - "hotpluggable", - "hotplugged", - "hv-apicv", - "hv-avic", - "hv-crash", - "hv-emsr-bitmap", - "hv-enforce-cpuid", - "hv-evmcs", - "hv-frequencies", - "hv-ipi", - "hv-passthrough", - "hv-reenlightenment", - "hv-relaxed", - "hv-reset", - "hv-runtime", - "hv-stimer", - "hv-stimer-direct", - "hv-syndbg", - "hv-synic", - "hv-time", - "hv-tlbflush", - "hv-tlbflush-direct", - "hv-tlbflush-ext", - "hv-vapic", - "hv-vpindex", - "hv-xmm-input", - "kvm", - "kvm-asyncpf", - "kvm-asyncpf-int", - "kvm-hint-dedicated", - "kvm-mmu", - "kvm-msi-ext-dest-id", - "kvm-no-smi-migration", - "kvm-nopiodelay", - "kvm-poll-control", - "kvm-pv-enforce-cpuid", - "kvm-pv-eoi", - "kvm-pv-ipi", - "kvm-pv-sched-yield", - "kvm-pv-tlb-flush", - "kvm-pv-unhalt", - "kvm-steal-time", - "kvm_asyncpf", - "kvm_asyncpf_int", - "kvm_mmu", - "kvm_nopiodelay", - "kvm_poll_control", - "kvm_pv_eoi", - "kvm_pv_unhalt", - "kvm_steal_time", - "kvmclock", - "kvmclock-stable-bit", - "l3-cache", - "legacy-cache", - "lmce", - "migratable", - "pmu", - "realized", - "start-powered-off", - "tcg-cpuid", - "vmware-cpuid-freq", - "xen-vapic", - ] - - if feature["type"] !=3D "bool": - return True - - name =3D feature["name"] - - if name.startswith("x-"): - return True - - if name in ignored_features: - return True - - return False - - -def get_qemu_feature_list(path_to_qemu): - cmd =3D [ - path_to_qemu, - "-machine", "accel=3Dkvm", - "-cpu", "host", - "-nodefaults", - "-nographic", - "-qmp", - "stdio" - ] - - request =3D """ - { - "execute": "qmp_capabilities" - } - { - "execute": "qom-list-properties", - "arguments": { - "typename": "max-x86_64-cpu" - }, - "id": "qom-list-properties" - } - { - "execute": "quit" +import os +import re + + +# mapping from qemu names to libvirt names for x86 feature names. +# a mapping to `None` will result in the feature being ignored. +FEATURE_NAMES =3D { + "cmp-legacy": "cmp_legacy", + "ds-cpl": "ds_cpl", + "fxsr-opt": "fxsr_opt", + "lahf-lm": "lahf_lm", + "nodeid-msr": "nodeid_msr", + "pclmulqdq": "pclmuldq", + "perfctr-core": "perfctr_core", + "perfctr-nb": "perfctr_nb", + "tsc-adjust": "tsc_adjust", + + "kvm-asyncpf": None, + "kvm-asyncpf-int": None, + "kvm-hint-dedicated": None, + "kvm-mmu": None, + "kvm-msi-ext-dest-id": None, + "kvm-nopiodelay": None, + "kvm-poll-control": None, + "kvm-pv-eoi": None, + "kvm-pv-ipi": None, + "kvm-pv-sched-yield": None, + "kvm-pv-tlb-flush": None, + "kvm-pv-unhalt": None, + "kvm-steal-time": None, + "kvmclock": None, + "kvmclock-stable-bit": None, + + "xstore": None, + "xstore-en": None, + "xcrypt": None, + "xcrypt-en": None, + "ace2": None, + "ace2-en": None, + "phe": None, + "phe-en": None, + "pmm": None, + "pmm-en": None, + + "full-width-write": None, +} + + +# features in libvirt, that qemu does not know. as python cannot use dicts +# as keys in other dicts, use tuples. three-tuples "eax, ecx, register nam= e" +# for cpuid features; one-tuples "index" for msrs. The values for the dict= are +# mappings from "bit index" to "feature name". +EXTRA_FEATURES =3D { + (0x00000001, None, "ecx"): { + 27: "osxsave", + }, + (0x00000007, 0x0000, "ebx"): { + 12: "cmt", + }, + (0x00000007, 0x0000, "ecx"): { + 4: "ospke", + }, + (0x00000007, 0x0000, "edx"): { + 18: "pconfig", + }, + (0x0000000f, 0x0001, "edx"): { + 1: "mbm_total", + 2: "mbm_local", + }, + (0x80000001, None, "ecx"): { + 18: "cvt16", + }, + (0x0000048c,): { + 8: "vmx-ept-uc", + 14: "vmx-ept-wb", + 41: "vmx-invvpid-single-context", # wrong name in qe= mu + 43: "vmx-invvpid-single-context-noglobals", # wrong name in qe= mu } - """ - - decoder =3D json.JSONDecoder() - output =3D subprocess.check_output(cmd, input=3Drequest, text=3DTrue) - while output: - obj, idx =3D decoder.raw_decode(output) - output =3D output[idx:].strip() - if obj.get("id") !=3D "qom-list-properties": +} + + +# list non-migratable features here +NON_MIGRATABLE_FEATURES =3D ( + "xsaves", + "invtsc", +) + + +# mapping from "symbol name" to "value" for "#define"s in qemu source code +_CONSTANTS =3D dict() + + +# tree of known features. top level index is either "cpuid" or "msr". +# further indices for cpuid: eax_in, ecx_in (may be `None`), register name +# further indices for msr: index +_FEATURES =3D dict() + + +# fill _CONSTANTS with the #defines from qemu source code +def read_headers(path): + pattern_define =3D re.compile("^#define\\s+(\\S+)\\s+(.*)$") + + headers =3D ( + "include/standard-headers/asm-x86/kvm_para.h", + "target/i386/cpu.h", + ) + + _CONSTANTS["true"] =3D "1" + + for header in headers: + with open(os.path.join(path, header), "tr") as f: + for line in f.readlines(): + match =3D pattern_define.match(line) + if match: + key =3D match.group(1) + val =3D match.group(2) + _CONSTANTS[key] =3D val + + +# add new cpuid feature bit +def add_feature_cpuid(eax, ecx, reg, bit, name): + if not name: + return + + if "cpuid" not in _FEATURES: + _FEATURES["cpuid"] =3D dict() + + if eax not in _FEATURES["cpuid"]: + _FEATURES["cpuid"][eax] =3D dict() + + if ecx not in _FEATURES["cpuid"][eax]: + _FEATURES["cpuid"][eax][ecx] =3D dict() + + if reg not in _FEATURES["cpuid"][eax][ecx]: + _FEATURES["cpuid"][eax][ecx][reg] =3D dict() + + _FEATURES["cpuid"][eax][ecx][reg][bit] =3D name + + +# add new msr feature bit +def add_feature_msr(msr, bit, name): + if not name: + return + + if "msr" not in _FEATURES: + _FEATURES["msr"] =3D dict() + + if msr not in _FEATURES["msr"]: + _FEATURES["msr"][msr] =3D dict() + + _FEATURES["msr"][msr][bit] =3D name + + +# add features from EXTRA_FEATURE to the list of known features +def add_extra_features(): + for key, val in EXTRA_FEATURES.items(): + for bit, name in val.items(): + if len(key) =3D=3D 3: + add_feature_cpuid(key[0], key[1], key[2], bit, name) + else: + add_feature_msr(key[0], bit, name) + + +# add a feature from qemu to the list of known features. translates featur= es +# names according to FEATURE_NAMES and applies symbolic values defined in +# _CONSTANTS. +def add_feature_raw(query, data): + # split names into individual items + data =3D [n.strip() for n in "".join(data).split(",")] + + names =3D dict() + if any([e.startswith("[") for e in data]): + for entry in data: + entry =3D entry.strip() + if not entry: + continue + index, name =3D entry.split("=3D", 2) + index =3D int(index.strip().strip("[").strip("]"), 0) + names[index] =3D name.strip().strip("\"") + else: + for index, name in enumerate(data): + if not name or name =3D=3D "NULL": + continue + names[index] =3D name.strip("\"") + + # cut out part between "{" and "}". easiest way to get rid of unwanted + # extra info such as ".tcg_features" or multi line comments + query =3D "".join(query).split("{")[1].split("}")[0] + + eax =3D None + ecx =3D None + reg =3D None + msr =3D None + for entry in [e.strip() for e in query.split(",")]: + if not entry: continue - for feature in obj["return"]: - if ignore_feature(feature): + left, right =3D [e.strip() for e in entry.split("=3D", 2)] + if left =3D=3D ".eax": + eax =3D int(_CONSTANTS.get(right, right), 0) + if left =3D=3D ".ecx": + ecx =3D int(_CONSTANTS.get(right, right), 0) + if left =3D=3D ".reg": + reg =3D right.lower()[2:] + if left =3D=3D ".index": + msr =3D int(_CONSTANTS.get(right, right), 0) + + # qemu defines some empty feature words, filter them out + if not names: + return + if all([e is None for e in names.values()]): + return + + # apply name translation and add to list of known features + for bit, name in sorted(names.items()): + name =3D FEATURE_NAMES.get(name, name) + if msr: + add_feature_msr(msr, bit, name) + else: + add_feature_cpuid(eax, ecx, reg, bit, name) + + +# read the `feature_word_info` struct from qemu's cpu.c into a list of str= ings +def read_cpu_c(path): + pattern_comment =3D re.compile("/\\*.*?\\*/") + marker_begin =3D "FeatureWordInfo feature_word_info[FEATURE_WORDS] =3D= {\n" + marker_end =3D "};\n" + + with open(os.path.join(path, "target/i386/cpu.c"), "tr") as f: + # skip until begin marker + while True: + line =3D f.readline() + if not line: + exit("begin marker not found in cpu.c") + if line =3D=3D marker_begin: + break + + # read until end marker + while True: + line =3D f.readline() + if not line: + exit("end marker not found in cpu.c") + if line =3D=3D marker_end: + break + + # remove comments and white space + line =3D re.sub(pattern_comment, "", line).strip() + + yield line + + +# simple state machine to extract feature names and definitions from extra= cted +# qemu source code +def parse_feature_words(lines): + state_waiting_for_type =3D 1 + state_waiting_for_names =3D 2 + state_read_names =3D 3 + state_waiting_for_query =3D 4 + state_read_query =3D 5 + + pattern_type =3D re.compile("^\\.type\\s*=3D\\s*(.+)$") + pattern_names =3D re.compile("^\\.feat_names\\s*=3D\\s*{$") + pattern_data =3D re.compile("^\\.(cpuid|msr).*$") + pattern_end =3D re.compile("^},?$") + + state =3D state_waiting_for_type + for line in lines: + if state =3D=3D state_waiting_for_type: + match =3D pattern_type.match(line) + if match: + data_names =3D list() + data_query =3D list() + state =3D state_waiting_for_names + + elif state =3D=3D state_waiting_for_names: + # special case for missing ".feat_names" entry: + match =3D pattern_data.match(line) + if match: + data_query.append(line) + state =3D state_read_query continue - yield feature["name"] =20 + match =3D pattern_names.match(line) + if match: + state =3D state_read_names + + elif state =3D=3D state_read_names: + match =3D pattern_end.match(line) + if match: + state =3D state_waiting_for_query + else: + data_names.append(line) + + elif state =3D=3D state_waiting_for_query: + match =3D pattern_data.match(line) + if match: + data_query.append(line) + state =3D state_read_query + + elif state =3D=3D state_read_query: + match =3D pattern_end.match(line) + data_query.append(line) + if match: + state =3D state_waiting_for_type + add_feature_raw(data_query, data_names) + + else: + exit("parsing state machine in invalid state") =20 -def get_libvirt_feature_list(path_to_featuresfile): - dom =3D xml.etree.ElementTree.parse(path_to_featuresfile) - for feature in dom.getroot().iter("feature"): - yield feature.get("name") - for alias in feature: - if alias.tag =3D=3D "alias" and alias.get("source") =3D=3D "qe= mu": - yield alias.get("name") + if state !=3D state_waiting_for_type: + exit("parsing incomplete") + + +# generate x86_features.xml from list of known features +def write_output(path): + with open(path, "tw") as f: + f.write("\n= \n") + f.write("\n") + + for eax in sorted(_FEATURES["cpuid"]): + for ecx in sorted(_FEATURES["cpuid"][eax]): + for reg in sorted(_FEATURES["cpuid"][eax][ecx]): + f.write(f"\n \n") + + names =3D sorted(_FEATURES["cpuid"][eax][ecx][reg].ite= ms()) + for bit, name in names: + mask =3D 1 << bit + f.write(f" \n") + f.write(f" \n") + f.write(" \n") + + for msr in sorted(_FEATURES["msr"]): + f.write(f"\n \n") + names =3D sorted(_FEATURES["msr"][msr].items()) + for bit, name in names: + mask =3D 1 << bit + f.write(f" \n") + f.write(f" > 32):08x}' ") + f.write(f"eax=3D'0x{(mask & 0xffffffff):08x}'/>\n") + f.write(" \n") + + f.write("\n") =20 =20 def main(): + dirname =3D os.path.dirname(__file__) + parser =3D argparse.ArgumentParser( - description=3D"Synchronize x86 cpu features from QEMU i386 target.= ") + description=3D"Synchronize x86 cpu features from QEMU." + ) parser.add_argument( - "--qemu", - help=3D"Path to qemu executable", - default=3D"qemu-system-x86_64", - type=3Dstr) + "qemu", + help=3D"Path to qemu source code", + default=3Dos.path.realpath(os.path.join(dirname, "../../../qemu")), + nargs=3D"?", + type=3Dos.path.realpath, + ) parser.add_argument( - "--features", - help=3D"Path to 'src/cpu_map/x86_features.xml' file in " - "the libvirt repository", - default=3D"x86_features.xml", - type=3Dstr) - + "--output", + "-o", + help=3D"Path to output file", + default=3Dos.path.realpath(os.path.join(dirname, "x86_features.xml= ")), + type=3Dos.path.realpath + ) args =3D parser.parse_args() =20 - qfeatures =3D get_qemu_feature_list(args.qemu) - lfeatures =3D list(get_libvirt_feature_list(args.features)) - missing =3D [f for f in sorted(qfeatures) if f not in lfeatures] + if not os.path.isdir(args.qemu): + parser.print_help() + exit("qemu source directory not found") =20 - if missing: - print("The following features were reported by qemu but are " - "unknown to libvirt:") - for feature in missing: - print(" *", feature) + read_headers(args.qemu) + lines =3D read_cpu_c(args.qemu) + parse_feature_words(lines) + add_extra_features() + write_output(args.output) =20 - return len(missing) !=3D 0 + print( + "After adding new features, update existing test files by running " + "`tests/cputestdata/cpu-data.py diff tests/cputestdata/" + "x86_64-cpuid-*.json`" + ) =20 =20 if __name__ =3D=3D "__main__": diff --git a/src/cpu_map/x86_features.xml b/src/cpu_map/x86_features.xml index 60c952223a..361fb5f18a 100644 --- a/src/cpu_map/x86_features.xml +++ b/src/cpu_map/x86_features.xml @@ -1,9 +1,8 @@ + =20 --=20 2.43.0 _______________________________________________ Devel mailing list -- devel@lists.libvirt.org To unsubscribe send an email to devel-leave@lists.libvirt.org