From nobody Fri Oct 18 05:24:20 2024 Delivered-To: importer@patchew.org Received-SPF: none (zohomail.com: 8.43.85.245 is neither permitted nor denied by domain of lists.libvirt.org) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; spf=none (zohomail.com: 8.43.85.245 is neither permitted nor denied by domain of lists.libvirt.org) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 1707130546253545.2014074103687; Mon, 5 Feb 2024 02:55:46 -0800 (PST) Received: by lists.libvirt.org (Postfix, from userid 996) id 2CCF91C5A; Mon, 5 Feb 2024 05:55:45 -0500 (EST) Received: from lists.libvirt.org.85.43.8.in-addr.arpa (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id C438E1D89; Mon, 5 Feb 2024 05:45:21 -0500 (EST) Received: by lists.libvirt.org (Postfix, from userid 996) id 3D1641BF1; Mon, 5 Feb 2024 05:44:34 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 2488B1B80 for ; Mon, 5 Feb 2024 05:44:31 -0500 (EST) Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-231-ga2Ytp5UMFCMMoDH5gkLdA-1; Mon, 05 Feb 2024 05:44:28 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7CBC61C06908 for ; Mon, 5 Feb 2024 10:44:28 +0000 (UTC) Received: from work.redhat.com (unknown [10.39.194.112]) by smtp.corp.redhat.com (Postfix) with ESMTP id 124A02026D66; Mon, 5 Feb 2024 10:44:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.4 X-MC-Unique: ga2Ytp5UMFCMMoDH5gkLdA-1 From: Tim Wiederhake To: devel@lists.libvirt.org Subject: [PATCH v2 4/4] cpu_map: Rewrite feature sync script Date: Mon, 5 Feb 2024 11:44:24 +0100 Message-ID: <20240205104424.80104-5-twiederh@redhat.com> In-Reply-To: <20240205104424.80104-1-twiederh@redhat.com> References: <20240205104424.80104-1-twiederh@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Message-ID-Hash: NQDVUIMJAPQVCYNADSNGD77CDDU65A75 X-Message-ID-Hash: NQDVUIMJAPQVCYNADSNGD77CDDU65A75 X-MailFrom: twiederh@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Tim Wiederhake X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="utf-8"; x-default="true" Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1707130546769100001 Previously, the script would only detect differences between libvirt's and qemu's list of x86 features, adding those features to libvirt was a manual and error prone procedure. Replace with a script that can generate libvirt's feature list directly from qemu source code. Reviewed-by: Jiri Denemark --- src/cpu_map/sync_qemu_features_i386.py | 627 ++++++++++++++++++------- src/cpu_map/x86_features.xml | 7 +- 2 files changed, 461 insertions(+), 173 deletions(-) diff --git a/src/cpu_map/sync_qemu_features_i386.py b/src/cpu_map/sync_qemu= _features_i386.py index dcd58cd590..e4b1f7275a 100755 --- a/src/cpu_map/sync_qemu_features_i386.py +++ b/src/cpu_map/sync_qemu_features_i386.py @@ -1,190 +1,479 @@ #!/usr/bin/env python3 =20 import argparse -import json -import subprocess -import xml.etree.ElementTree - - -def ignore_feature(feature): - ignored_features =3D [ - # VIA/Cyrix/Centaur-defined CPU features - # CPUID level 0xC0000001, word 5 - "ace2", - "ace2-en", - "phe", - "phe-en", - "pmm", - "pmm-en", - "xcrypt", - "xcrypt-en", - "xstore", - "xstore-en", - - # non-features - "check", - "cpuid-0xb", - "enforce", - "fill-mtrr-mask", - "full-cpuid-auto-level", - "full-width-write", - "host-cache-info", - "host-phys-bits", - "hotpluggable", - "hotplugged", - "hv-apicv", - "hv-avic", - "hv-crash", - "hv-emsr-bitmap", - "hv-enforce-cpuid", - "hv-evmcs", - "hv-frequencies", - "hv-ipi", - "hv-passthrough", - "hv-reenlightenment", - "hv-relaxed", - "hv-reset", - "hv-runtime", - "hv-stimer", - "hv-stimer-direct", - "hv-syndbg", - "hv-synic", - "hv-time", - "hv-tlbflush", - "hv-tlbflush-direct", - "hv-tlbflush-ext", - "hv-vapic", - "hv-vpindex", - "hv-xmm-input", - "kvm", - "kvm-asyncpf", - "kvm-asyncpf-int", - "kvm-hint-dedicated", - "kvm-mmu", - "kvm-msi-ext-dest-id", - "kvm-no-smi-migration", - "kvm-nopiodelay", - "kvm-poll-control", - "kvm-pv-enforce-cpuid", - "kvm-pv-eoi", - "kvm-pv-ipi", - "kvm-pv-sched-yield", - "kvm-pv-tlb-flush", - "kvm-pv-unhalt", - "kvm-steal-time", - "kvm_asyncpf", - "kvm_asyncpf_int", - "kvm_mmu", - "kvm_nopiodelay", - "kvm_poll_control", - "kvm_pv_eoi", - "kvm_pv_unhalt", - "kvm_steal_time", - "kvmclock", - "kvmclock-stable-bit", - "l3-cache", - "legacy-cache", - "lmce", - "migratable", - "pmu", - "realized", - "start-powered-off", - "tcg-cpuid", - "vmware-cpuid-freq", - "xen-vapic", - ] - - if feature["type"] !=3D "bool": - return True - - name =3D feature["name"] - - if name.startswith("x-"): - return True - - if name in ignored_features: - return True - - return False - - -def get_qemu_feature_list(path_to_qemu): - cmd =3D [ - path_to_qemu, - "-machine", "accel=3Dkvm", - "-cpu", "host", - "-nodefaults", - "-nographic", - "-qmp", - "stdio" - ] - - request =3D """ - { - "execute": "qmp_capabilities" - } - { - "execute": "qom-list-properties", - "arguments": { - "typename": "max-x86_64-cpu" - }, - "id": "qom-list-properties" - } - { - "execute": "quit" +import os +import re + + +# features in qemu that we do not want in libvirt +FEATURES_IGNORE =3D ( + "kvm-asyncpf", + "kvm-asyncpf-int", + "kvm-hint-dedicated", + "kvm-mmu", + "kvm-msi-ext-dest-id", + "kvm-nopiodelay", + "kvm-poll-control", + "kvm-pv-eoi", + "kvm-pv-ipi", + "kvm-pv-sched-yield", + "kvm-pv-tlb-flush", + "kvm-pv-unhalt", + "kvm-steal-time", + "kvmclock", + "kvmclock-stable-bit", + + "xstore", + "xstore-en", + "xcrypt", + "xcrypt-en", + "ace2", + "ace2-en", + "phe", + "phe-en", + "pmm", + "pmm-en", + + "full-width-write", +) + + +# features in libvirt, that qemu does not know. as python cannot use dicts +# as keys in other dicts, use tuples. three-tuples "eax, ecx, register nam= e" +# for cpuid features; one-tuples "index" for msrs. The values for the dict= are +# mappings from "bit index" to "feature name". +FEATURES_EXTRA =3D { + (0x00000001, None, "ecx"): { + 27: "osxsave", + }, + (0x00000007, 0x0000, "ebx"): { + 12: "cmt", + }, + (0x00000007, 0x0000, "ecx"): { + 4: "ospke", + }, + (0x00000007, 0x0000, "edx"): { + 18: "pconfig", + }, + (0x0000000f, 0x0001, "edx"): { + 1: "mbm_total", + 2: "mbm_local", + }, + (0x80000001, None, "ecx"): { + 18: "cvt16", + }, + (0x0000048c,): { + 8: "vmx-ept-uc", + 14: "vmx-ept-wb", + 41: "vmx-invvpid-single-context", # wrong name in qe= mu + 43: "vmx-invvpid-single-context-noglobals", # wrong name in qe= mu } - """ - - decoder =3D json.JSONDecoder() - output =3D subprocess.check_output(cmd, input=3Drequest, text=3DTrue) - while output: - obj, idx =3D decoder.raw_decode(output) - output =3D output[idx:].strip() - if obj.get("id") !=3D "qom-list-properties": +} + + +# alias information to add to generated file +FEATURES_ALIASES =3D { + "arch-capabilities": ( + ("arch_capabilities", "linux"), + ), + "cmp_legacy": ( + ("cmp-legacy", "qemu"), + ), + "cmt": ( + ("cqm", "linux"), + ), + "ds_cpl": ( + ("ds-cpl", "qemu"), + ), + "fxsr_opt": ( + ("ffxsr", "qemu"), + ("fxsr-opt", "qemu"), + ), + "lahf_lm": ( + ("lahf-lm", "qemu"), + ), + "lm": ( + ("i64", "qemu"), + ), + "md-clear": ( + ("md_clear", "linux"), + ), + "nodeid_msr": ( + ("nodeid-msr", "qemu"), + ), + "nrip-save": ( + ("nrip_save", "qemu"), + ), + "nx": ( + ("xd", "qemu"), + ), + "pause-filter": ( + ("pause_filter", "qemu"), + ), + "pclmuldq": ( + ("pclmulqdq", "qemu"), + ), + "perfctr_core": ( + ("perfctr-core", "qemu"), + ), + "perfctr_nb": ( + ("perfctr-nb", "qemu"), + ), + "pni": ( + ("sse3", "qemu"), + ), + "sse4.1": ( + ("sse4-1", "qemu"), + ("sse4_1", "qemu"), + ), + "sse4.2": ( + ("sse4-2", "qemu"), + ("sse4_2", "qemu"), + ), + "svm-lock": ( + ("svm_lock", "qemu"), + ), + "tsc-scale": ( + ("tsc_scale", "qemu"), + ), + "tsc_adjust": ( + ("tsc-adjust", "qemu"), + ), + "vmcb-clean": ( + ("vmcb_clean", "qemu"), + ), + "vmx-invvpid-single-context-noglobals": ( + ("vmx-invept-single-context-noglobals", "qemu"), + ), +} + + +# list non-migratable features here +FEATURES_NON_MIGRATABLE =3D ( + "xsaves", + "invtsc", +) + + +# mapping from "symbol name" to "value" for "#define"s in qemu source code +_CONSTANTS =3D dict() + + +# tree of known features. top level index is either "cpuid" or "msr". +# further indices for cpuid: eax_in, ecx_in (may be `None`), register name +# further indices for msr: index +_FEATURES =3D dict() + + +# fill _CONSTANTS with the #defines from qemu source code +def read_headers(path): + pattern_define =3D re.compile("^#define\\s+(\\S+)\\s+(.*)$") + + headers =3D ( + "include/standard-headers/asm-x86/kvm_para.h", + "target/i386/cpu.h", + ) + + _CONSTANTS["true"] =3D "1" + + for header in headers: + with open(os.path.join(path, header), "tr") as f: + for line in f.readlines(): + match =3D pattern_define.match(line) + if match: + key =3D match.group(1) + val =3D match.group(2) + _CONSTANTS[key] =3D val + + +# add new cpuid feature bit +def add_feature_cpuid(eax, ecx, reg, bit, name): + if not name: + return + + if "cpuid" not in _FEATURES: + _FEATURES["cpuid"] =3D dict() + + if eax not in _FEATURES["cpuid"]: + _FEATURES["cpuid"][eax] =3D dict() + + if ecx not in _FEATURES["cpuid"][eax]: + _FEATURES["cpuid"][eax][ecx] =3D dict() + + if reg not in _FEATURES["cpuid"][eax][ecx]: + _FEATURES["cpuid"][eax][ecx][reg] =3D dict() + + _FEATURES["cpuid"][eax][ecx][reg][bit] =3D name + + +# add new msr feature bit +def add_feature_msr(msr, bit, name): + if not name: + return + + if "msr" not in _FEATURES: + _FEATURES["msr"] =3D dict() + + if msr not in _FEATURES["msr"]: + _FEATURES["msr"][msr] =3D dict() + + _FEATURES["msr"][msr][bit] =3D name + + +# add features from EXTRA_FEATURE to the list of known features +def add_extra_features(): + for key, val in FEATURES_EXTRA.items(): + for bit, name in val.items(): + if len(key) =3D=3D 3: + add_feature_cpuid(key[0], key[1], key[2], bit, name) + else: + add_feature_msr(key[0], bit, name) + + +# add a feature from qemu to the list of known features. translates featur= es +# names according to FEATURE_ALIASES and applies symbolic values defined in +# _CONSTANTS. +def add_feature_qemu(query, data): + # split names into individual items + data =3D [n.strip() for n in "".join(data).split(",")] + + names =3D dict() + if any([e.startswith("[") for e in data]): + for entry in data: + entry =3D entry.strip() + if not entry: + continue + index, name =3D entry.split("=3D", 2) + index =3D int(index.strip().strip("[").strip("]"), 0) + names[index] =3D name.strip().strip("\"") + else: + for index, name in enumerate(data): + if not name or name =3D=3D "NULL": + continue + name =3D name.strip("\"") + if name in FEATURES_IGNORE: + continue + names[index] =3D name.strip("\"") + + # cut out part between "{" and "}". easiest way to get rid of unwanted + # extra info such as ".tcg_features" or multi line comments + query =3D "".join(query).split("{")[1].split("}")[0] + + eax =3D None + ecx =3D None + reg =3D None + msr =3D None + for entry in [e.strip() for e in query.split(",")]: + if not entry: continue - for feature in obj["return"]: - if ignore_feature(feature): + left, right =3D [e.strip() for e in entry.split("=3D", 2)] + if left =3D=3D ".eax": + eax =3D int(_CONSTANTS.get(right, right), 0) + if left =3D=3D ".ecx": + ecx =3D int(_CONSTANTS.get(right, right), 0) + if left =3D=3D ".reg": + reg =3D right.lower()[2:] + if left =3D=3D ".index": + msr =3D int(_CONSTANTS.get(right, right), 0) + + # qemu defines some empty feature words, filter them out + if not names: + return + if all([e is None for e in names.values()]): + return + + # apply name translation and add to list of known features + for bit, name in sorted(names.items()): + for newname, data in FEATURES_ALIASES.items(): + for oldname, source in data: + if name =3D=3D oldname and source =3D=3D "qemu": + name =3D newname + + if msr: + add_feature_msr(msr, bit, name) + else: + add_feature_cpuid(eax, ecx, reg, bit, name) + + +# read the `feature_word_info` struct from qemu's cpu.c into a list of str= ings +def read_cpu_c(path): + pattern_comment =3D re.compile("/\\*.*?\\*/") + marker_begin =3D "FeatureWordInfo feature_word_info[FEATURE_WORDS] =3D= {\n" + marker_end =3D "};\n" + + with open(os.path.join(path, "target/i386/cpu.c"), "tr") as f: + # skip until begin marker + while True: + line =3D f.readline() + if not line: + exit("begin marker not found in cpu.c") + if line =3D=3D marker_begin: + break + + # read until end marker + while True: + line =3D f.readline() + if not line: + exit("end marker not found in cpu.c") + if line =3D=3D marker_end: + break + + # remove comments and white space + line =3D re.sub(pattern_comment, "", line).strip() + + yield line + + +# simple state machine to extract feature names and definitions from extra= cted +# qemu source code +def parse_feature_words(lines): + state_waiting_for_type =3D 1 + state_waiting_for_names =3D 2 + state_read_names =3D 3 + state_waiting_for_query =3D 4 + state_read_query =3D 5 + + pattern_type =3D re.compile("^\\.type\\s*=3D\\s*(.+)$") + pattern_names =3D re.compile("^\\.feat_names\\s*=3D\\s*{$") + pattern_data =3D re.compile("^\\.(cpuid|msr).*$") + pattern_end =3D re.compile("^},?$") + + state =3D state_waiting_for_type + for line in lines: + if state =3D=3D state_waiting_for_type: + match =3D pattern_type.match(line) + if match: + data_names =3D list() + data_query =3D list() + state =3D state_waiting_for_names + + elif state =3D=3D state_waiting_for_names: + # special case for missing ".feat_names" entry: + match =3D pattern_data.match(line) + if match: + data_query.append(line) + state =3D state_read_query continue - yield feature["name"] =20 + match =3D pattern_names.match(line) + if match: + state =3D state_read_names + + elif state =3D=3D state_read_names: + match =3D pattern_end.match(line) + if match: + state =3D state_waiting_for_query + else: + data_names.append(line) + + elif state =3D=3D state_waiting_for_query: + match =3D pattern_data.match(line) + if match: + data_query.append(line) + state =3D state_read_query + + elif state =3D=3D state_read_query: + match =3D pattern_end.match(line) + data_query.append(line) + if match: + state =3D state_waiting_for_type + add_feature_qemu(data_query, data_names) + + else: + exit("parsing state machine in invalid state") =20 -def get_libvirt_feature_list(path_to_featuresfile): - dom =3D xml.etree.ElementTree.parse(path_to_featuresfile) - for feature in dom.getroot().iter("feature"): - yield feature.get("name") - for alias in feature: - if alias.tag =3D=3D "alias" and alias.get("source") =3D=3D "qe= mu": - yield alias.get("name") + if state !=3D state_waiting_for_type: + exit("parsing incomplete") + + +# generate x86_features.xml from list of known features +def write_output(path): + with open(path, "tw") as f: + f.write("\n= \n") + f.write("\n") + + for eax in sorted(_FEATURES["cpuid"]): + for ecx in sorted(_FEATURES["cpuid"][eax]): + for reg in sorted(_FEATURES["cpuid"][eax][ecx]): + f.write(f"\n \n") + + names =3D sorted(_FEATURES["cpuid"][eax][ecx][reg].ite= ms()) + for bit, name in names: + mask =3D 1 << bit + f.write(f" \n") + for alias in FEATURES_ALIASES.get(name, []): + f.write(f" \n") + f.write(f" \n") + f.write(" \n") + + for msr in sorted(_FEATURES["msr"]): + f.write(f"\n \n") + names =3D sorted(_FEATURES["msr"][msr].items()) + for bit, name in names: + mask =3D 1 << bit + f.write(f" \n") + for alias in FEATURES_ALIASES.get(name, []): + f.write(f" \n") + f.write(f" > 32):08x}' ") + f.write(f"eax=3D'0x{(mask & 0xffffffff):08x}'/>\n") + f.write(" \n") + + f.write("\n") =20 =20 def main(): + dirname =3D os.path.dirname(__file__) + parser =3D argparse.ArgumentParser( - description=3D"Synchronize x86 cpu features from QEMU i386 target.= ") + description=3D"Synchronize x86 cpu features from QEMU." + ) parser.add_argument( - "--qemu", - help=3D"Path to qemu executable", - default=3D"qemu-system-x86_64", - type=3Dstr) + "qemu", + help=3D"Path to qemu source code", + default=3Dos.path.realpath(os.path.join(dirname, "../../../qemu")), + nargs=3D"?", + type=3Dos.path.realpath, + ) parser.add_argument( - "--features", - help=3D"Path to 'src/cpu_map/x86_features.xml' file in " - "the libvirt repository", - default=3D"x86_features.xml", - type=3Dstr) - + "--output", + "-o", + help=3D"Path to output file", + default=3Dos.path.realpath(os.path.join(dirname, "x86_features.xml= ")), + type=3Dos.path.realpath + ) args =3D parser.parse_args() =20 - qfeatures =3D get_qemu_feature_list(args.qemu) - lfeatures =3D list(get_libvirt_feature_list(args.features)) - missing =3D [f for f in sorted(qfeatures) if f not in lfeatures] + if not os.path.isdir(args.qemu): + parser.print_help() + exit("qemu source directory not found") =20 - if missing: - print("The following features were reported by qemu but are " - "unknown to libvirt:") - for feature in missing: - print(" *", feature) + read_headers(args.qemu) + lines =3D read_cpu_c(args.qemu) + parse_feature_words(lines) + add_extra_features() + write_output(args.output) =20 - return len(missing) !=3D 0 + print( + "After adding new features, update existing test files by running " + "`tests/cputestdata/cpu-data.py diff tests/cputestdata/" + "x86_64-cpuid-*.json`" + ) =20 =20 if __name__ =3D=3D "__main__": diff --git a/src/cpu_map/x86_features.xml b/src/cpu_map/x86_features.xml index 8d1dbe68e8..da71a3a6a0 100644 --- a/src/cpu_map/x86_features.xml +++ b/src/cpu_map/x86_features.xml @@ -1,9 +1,8 @@ + =20 --=20 2.43.0 _______________________________________________ Devel mailing list -- devel@lists.libvirt.org To unsubscribe send an email to devel-leave@lists.libvirt.org