From nobody Mon Feb 9 09:32:24 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1604487249; cv=none; d=zohomail.com; s=zohoarc; b=JF3A0uossXhjUXOGIflO97XuQ51a9rs1FGA5kwU8R9oyudUf8O+NQrZ8StDy3n2AMKOl8aGVfeC4hh7jWxDF5wSFZaFbPy5+mMNsM6q9Ff36y0O6q6UqPO2oJFwlXc3AYiX+gcZ4lwSkaypK8tWHuZp5+dzk5i1/BOuzjd4JAw4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1604487249; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=3akDq4D7ASNME6c9EEq8d+Ibum/hkwTpBKpg4yQPGaY=; b=oIHdaQZsO/JDnsJb9WLEx7IfG+RSNFqPJIjjXnpbWiC8a+67dDQsyW6N01cHEyAxwtEUd7eEwAD4mGMD6sxKoU2Luk+0umWseVRkKpnkPWrOeOEYukBiUCqgw4z5Pv1xAx4rHd5CGiN6Y/RPZEqnCsG852RiMAu+FrManRIHQy8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1604487249230535.1438267612335; Wed, 4 Nov 2020 02:54:09 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-240-qpmCuU7XPQe-Ajhrr_7DgQ-1; Wed, 04 Nov 2020 05:54:05 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 731501891E84; Wed, 4 Nov 2020 10:53:59 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0B78D5B4DE; Wed, 4 Nov 2020 10:53:59 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id A0A3658123; Wed, 4 Nov 2020 10:53:58 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 0A4Arv2l014044 for ; Wed, 4 Nov 2020 05:53:57 -0500 Received: by smtp.corp.redhat.com (Postfix) id 3F5C96EF46; Wed, 4 Nov 2020 10:53:57 +0000 (UTC) Received: from work.redhat.com (ovpn-112-204.ams2.redhat.com [10.36.112.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 516D473667; Wed, 4 Nov 2020 10:53:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604487248; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=3akDq4D7ASNME6c9EEq8d+Ibum/hkwTpBKpg4yQPGaY=; b=NGP9wH01Jgu/pmY52b6NVk/lfa5d6gSN5oEz6H/Zg36NqR5BpFukvKdmLG82eo6sFiQ897 9f6DOG3XuXHQ+4JVefIOgVufO4oWDc20y/G6b3op4if6mCSzEX4ZbCx7l3Ci8Ez0CrcgA8 ZnaNeC8nut+5sJ9YTLiJ4VeE9t9Usec= X-MC-Unique: qpmCuU7XPQe-Ajhrr_7DgQ-1 From: Tim Wiederhake To: libvir-list@redhat.com Subject: [libvirt PATCH v3 1/2] cpu_map: Add script to sync from QEMU i386 cpu models Date: Wed, 4 Nov 2020 11:53:50 +0100 Message-Id: <20201104105351.77383-2-twiederh@redhat.com> In-Reply-To: <20201104105351.77383-1-twiederh@redhat.com> References: <20201104105351.77383-1-twiederh@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Cc: Tim Wiederhake X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This script is intended to help in synchronizing i386 QEMU cpu model definitions with libvirt. As the QEMU cpu model definitions are post processed by QEMU and not meant to be consumed by third parties directly, parsing this information is imperfect. Additionally, the libvirt models contain information that cannot be generated from the QEMU data, preventing fully automated usage. The output should nevertheless be helpful for a human in determining potentially interesting changes. Signed-off-by: Tim Wiederhake Reviewed-by: Jiri Denemark --- src/cpu_map/sync_qemu_i386.py | 369 ++++++++++++++++++++++++++++++++++ 1 file changed, 369 insertions(+) create mode 100755 src/cpu_map/sync_qemu_i386.py diff --git a/src/cpu_map/sync_qemu_i386.py b/src/cpu_map/sync_qemu_i386.py new file mode 100755 index 0000000000..8deda869df --- /dev/null +++ b/src/cpu_map/sync_qemu_i386.py @@ -0,0 +1,369 @@ +#!/usr/bin/env python3 + +import argparse +import copy +import lark +import os +import re + + +T =3D { + # translating qemu -> libvirt cpu vendor names + "CPUID_VENDOR_AMD": "AMD", + "CPUID_VENDOR_INTEL": "Intel", + "CPUID_VENDOR_HYGON": "Hygon", + + # translating qemu -> libvirt cpu feature names + "CPUID_6_EAX_ARAT": "arat", + "CPUID_7_0_EBX_ADX": "adx", + "CPUID_7_0_EBX_AVX2": "avx2", + "CPUID_7_0_EBX_AVX512BW": "avx512bw", + "CPUID_7_0_EBX_AVX512CD": "avx512cd", + "CPUID_7_0_EBX_AVX512DQ": "avx512dq", + "CPUID_7_0_EBX_AVX512ER": "avx512er", + "CPUID_7_0_EBX_AVX512F": "avx512f", + "CPUID_7_0_EBX_AVX512PF": "avx512pf", + "CPUID_7_0_EBX_AVX512VL": "avx512vl", + "CPUID_7_0_EBX_BMI1": "bmi1", + "CPUID_7_0_EBX_BMI2": "bmi2", + "CPUID_7_0_EBX_CLFLUSHOPT": "clflushopt", + "CPUID_7_0_EBX_CLWB": "clwb", + "CPUID_7_0_EBX_ERMS": "erms", + "CPUID_7_0_EBX_FSGSBASE": "fsgsbase", + "CPUID_7_0_EBX_HLE": "hle", + "CPUID_7_0_EBX_INVPCID": "invpcid", + "CPUID_7_0_EBX_MPX": "mpx", + "CPUID_7_0_EBX_RDSEED": "rdseed", + "CPUID_7_0_EBX_RTM": "rtm", + "CPUID_7_0_EBX_SHA_NI": "sha-ni", + "CPUID_7_0_EBX_SMAP": "smap", + "CPUID_7_0_EBX_SMEP": "smep", + "CPUID_7_0_ECX_AVX512BITALG": "avx512bitalg", + "CPUID_7_0_ECX_AVX512_VBMI2": "avx512vbmi2", + "CPUID_7_0_ECX_AVX512_VBMI": "avx512vbmi", + "CPUID_7_0_ECX_AVX512VNNI": "avx512vnni", + "CPUID_7_0_ECX_AVX512_VPOPCNTDQ": "avx512-vpopcntdq", + "CPUID_7_0_ECX_CLDEMOTE": "cldemote", + "CPUID_7_0_ECX_GFNI": "gfni", + "CPUID_7_0_ECX_LA57": "la57", + "CPUID_7_0_ECX_MOVDIR64B": "movdir64b", + "CPUID_7_0_ECX_MOVDIRI": "movdiri", + "CPUID_7_0_ECX_PKU": "pku", + "CPUID_7_0_ECX_RDPID": "rdpid", + "CPUID_7_0_ECX_UMIP": "umip", + "CPUID_7_0_ECX_VAES": "vaes", + "CPUID_7_0_ECX_VPCLMULQDQ": "vpclmulqdq", + "CPUID_7_0_EDX_ARCH_CAPABILITIES": "arch-capabilities", + "CPUID_7_0_EDX_AVX512_4FMAPS": "avx512-4fmaps", + "CPUID_7_0_EDX_AVX512_4VNNIW": "avx512-4vnniw", + "CPUID_7_0_EDX_CORE_CAPABILITY": "core-capability", + "CPUID_7_0_EDX_SPEC_CTRL": "spec-ctrl", + "CPUID_7_0_EDX_SPEC_CTRL_SSBD": "ssbd", + "CPUID_7_0_EDX_STIBP": "stibp", + "CPUID_7_1_EAX_AVX512_BF16": "avx512-bf16", + "CPUID_8000_0008_EBX_CLZERO": "clzero", + "CPUID_8000_0008_EBX_IBPB": "ibpb", + "CPUID_8000_0008_EBX_STIBP": "amd-stibp", + "CPUID_8000_0008_EBX_WBNOINVD": "wbnoinvd", + "CPUID_8000_0008_EBX_XSAVEERPTR": "xsaveerptr", + "CPUID_ACPI": "acpi", + "CPUID_APIC": "apic", + "CPUID_CLFLUSH": "clflush", + "CPUID_CMOV": "cmov", + "CPUID_CX8": "cx8", + "CPUID_DE": "de", + "CPUID_EXT2_3DNOW": "3dnow", + "CPUID_EXT2_3DNOWEXT": "3dnowext", + "CPUID_EXT2_FFXSR": "fxsr_opt", + "CPUID_EXT2_LM": "lm", + "CPUID_EXT2_MMXEXT": "mmxext", + "CPUID_EXT2_NX": "nx", + "CPUID_EXT2_PDPE1GB": "pdpe1gb", + "CPUID_EXT2_RDTSCP": "rdtscp", + "CPUID_EXT2_SYSCALL": "syscall", + "CPUID_EXT3_3DNOWPREFETCH": "3dnowprefetch", + "CPUID_EXT3_ABM": "abm", + "CPUID_EXT3_CR8LEG": "cr8legacy", + "CPUID_EXT3_FMA4": "fma4", + "CPUID_EXT3_LAHF_LM": "lahf_lm", + "CPUID_EXT3_MISALIGNSSE": "misalignsse", + "CPUID_EXT3_OSVW": "osvw", + "CPUID_EXT3_PERFCORE": "perfctr_core", + "CPUID_EXT3_SSE4A": "sse4a", + "CPUID_EXT3_SVM": "svm", + "CPUID_EXT3_TBM": "tbm", + "CPUID_EXT3_XOP": "xop", + "CPUID_EXT_AES": "aes", + "CPUID_EXT_AVX": "avx", + "CPUID_EXT_CX16": "cx16", + "CPUID_EXT_F16C": "f16c", + "CPUID_EXT_FMA": "fma", + "CPUID_EXT_MOVBE": "movbe", + "CPUID_EXT_PCID": "pcid", + "CPUID_EXT_PCLMULQDQ": "pclmuldq", + "CPUID_EXT_POPCNT": "popcnt", + "CPUID_EXT_RDRAND": "rdrand", + "CPUID_EXT_SSE3": "pni", + "CPUID_EXT_SSE41": "sse4.1", + "CPUID_EXT_SSE42": "sse4.2", + "CPUID_EXT_SSSE3": "ssse3", + "CPUID_EXT_TSC_DEADLINE_TIMER": "tsc-deadline", + "CPUID_EXT_X2APIC": "x2apic", + "CPUID_EXT_XSAVE": "xsave", + "CPUID_FP87": "fpu", + "CPUID_FXSR": "fxsr", + "CPUID_MCA": "mca", + "CPUID_MCE": "mce", + "CPUID_MMX": "mmx", + "CPUID_MSR": "msr", + "CPUID_MTRR": "mtrr", + "CPUID_PAE": "pae", + "CPUID_PAT": "pat", + "CPUID_PGE": "pge", + "CPUID_PSE36": "pse36", + "CPUID_PSE": "pse", + "CPUID_SEP": "sep", + "CPUID_SSE2": "sse2", + "CPUID_SSE": "sse", + "CPUID_SS": "ss", + "CPUID_SVM_NPT": "npt", + "CPUID_SVM_NRIPSAVE": "nrip-save", + "CPUID_TSC": "tsc", + "CPUID_VME": "vme", + "CPUID_XSAVE_XGETBV1": "xgetbv1", + "CPUID_XSAVE_XSAVEC": "xsavec", + "CPUID_XSAVE_XSAVEOPT": "xsaveopt", + "CPUID_XSAVE_XSAVES": "xsaves", + "MSR_ARCH_CAP_IBRS_ALL": "ibrs-all", + "MSR_ARCH_CAP_MDS_NO": "mds-no", + "MSR_ARCH_CAP_PSCHANGE_MC_NO": "pschange-mc-no", + "MSR_ARCH_CAP_RDCL_NO": "rdctl-no", + "MSR_ARCH_CAP_SKIP_L1DFL_VMENTRY": "skip-l1dfl-vmentry", + "MSR_ARCH_CAP_TAA_NO": "taa-no", + "MSR_CORE_CAP_SPLIT_LOCK_DETECT": "split-lock-detect", + + # always disabled features + "CPUID_EXT_MONITOR": None, + "0": None, + + # set to "no auto enable" by qemu + "CPUID_EXT3_TOPOEXT": None, + "MSR_VMX_BASIC_DUAL_MONITOR": None, +} + + +def readline_cont(f): + """Read one logical line from a file `f` i.e. continues lines that end= in + a backslash.""" + + line =3D f.readline() + while line.endswith("\\\n"): + line =3D line[:-2] + " " + f.readline() + return line + + +def read_builtin_x86_defs(filename): + """Extract content between begin_mark and end_mark from file `filename= ` as + string, while expanding shorthand macros like "I486_FEATURES".""" + + begin_mark =3D "static X86CPUDefinition builtin_x86_defs[] =3D {\n" + end_mark =3D "};\n" + shorthand =3D re.compile("^#define ([A-Z0-9_]+_FEATURES) (.*)$") + lines =3D list() + shorthands =3D dict() + + with open(filename, "rt") as f: + while True: + line =3D readline_cont(f) + if line =3D=3D begin_mark: + break + if not line: + raise RuntimeError("begin mark not found") + match =3D shorthand.match(line) + if match: + # TCG definitions are irrelevant for cpu models + newk =3D match.group(1) + if newk.startswith("TCG_"): + continue + + # remove comments, whitespace and bit operators, effective= ly + # turning the bitfield into a list + newv =3D re.sub("([()|\t\n])|(/\\*.*?\\*/)", " ", match.gr= oup(2)) + + # resolve recursive shorthands + for k, v in shorthands.items(): + newv =3D newv.replace(k, v) + + shorthands[newk] =3D newv + + while True: + line =3D readline_cont(f) + if line =3D=3D end_mark: + break + if not line: + raise RuntimeError("end marker not found") + + # apply shorthands + for k, v in shorthands.items(): + line =3D line.replace(k, v) + lines.append(line) + + return "".join(lines) + + +def transform(item): + """Recursively transform a Lark syntax tree into python native objects= .""" + + if isinstance(item, lark.lexer.Token): + return str(item) + + if item.data =3D=3D "list": + retval =3D list() + for child in item.children: + value =3D transform(child) + if value is None: + continue + retval.append(value) + return retval + + if item.data =3D=3D "map": + retval =3D dict() + for child in item.children: + if len(child.children) !=3D 2: + raise RuntimeError("map entry with more than 2 elements") + key =3D transform(child.children[0]) + value =3D transform(child.children[1]) + if key is None: + raise RuntimeError("map entry with 'None' key") + if value is None: + continue + retval[key] =3D value + return retval + + if item.data =3D=3D "text": + retval =3D list() + for child in item.children: + value =3D transform(child) + if value is None: + continue + retval.append(value) + return " ".join(retval) + + if item.data =3D=3D "value": + if item.children: + raise RuntimeError("empty list is not empty") + return None + + raise RuntimeError("unexpected item type") + + +def expand_model(model): + """Expand a qemu cpu model description that has its feature split up i= nto + different fields and may have differing versions into several libvirt- + friendly cpu models.""" + + result =3D { + "name": model.pop(".name"), + "vendor": T[model.pop(".vendor")], + "features": set(), + "extra": dict()} + + if ".family" in model and ".model" in model: + result["family"] =3D model.pop(".family") + result["model"] =3D model.pop(".model") + + for k in [k for k in model if k.startswith(".features")]: + v =3D model.pop(k) + for feature in v.split(): + if feature.startswith("VMX_") or feature.startswith("MSR_VMX_"= ): + continue + translated =3D T.get(feature, feature) + if translated: + result["features"].add(translated) + + versions =3D model.pop(".versions", []) + for k, v in model.items(): + result["extra"]["model" + k] =3D v + yield result + + for version in versions: + result =3D copy.deepcopy(result) + result["name"] =3D version.pop(".alias", result["name"]) + + props =3D version.pop(".props", dict()) + for k, v in props: + if v =3D=3D "on": + result["features"].add(k) + elif v =3D=3D "off" and k in result["features"]: + result["features"].remove(k) + else: + result["extra"]["property." + k] =3D v + + for k, v in version.items(): + result["extra"]["version" + k] =3D v + + yield result + + +def output_model(f, model): + if model["extra"]: + f.write("\n") + + f.write("\n") + f.write(" \n".format(model["name"])) + f.write(" \n") + f.write(" \n".format( + model["family"], model["model"])) + f.write(" \n".format(model["vendor"])) + for feature in sorted(model["features"]): + f.write(" \n".format(feature)) + f.write(" \n") + f.write("\n") + + +def main(): + parser =3D argparse.ArgumentParser( + description=3D"Synchronize x86 cpu models from QEMU i386 target.") + parser.add_argument( + "cpufile", + help=3D"Path to 'target/i386/cpu.c' file in the QEMU repository", + type=3Dos.path.realpath) + parser.add_argument( + "outdir", + help=3D"Path to 'src/cpu_map' directory in the libvirt repository", + type=3Dos.path.realpath) + + args =3D parser.parse_args() + + builtin_x86_defs =3D read_builtin_x86_defs(args.cpufile) + + ast =3D lark.Lark(r""" + list: value ( "," value )* ","? + map: keyvalue ( "," keyvalue )* ","? + keyvalue: IDENTIFIER "=3D" value + ?value: text | "{" "}" | "{" list "}" | "{" map "}" + text: (IDENTIFIER | "\"" (/[^"]+/)? "\"")+ + IDENTIFIER: /[\[\]\._&a-zA-Z0-9]/+ + %ignore (" " | "\r" | "\n" | "\t" | "|" )+ + %ignore "(" ( "X86CPUVersionDefinition" | "PropValue" ) "[])" + %ignore "//" /.*?/ "\n" + %ignore "/*" /(.|\n)*?/ "*/" + """, start=3D"list").parse(builtin_x86_defs) + + models_json =3D transform(ast) + + models =3D list() + for model in models_json: + models.extend(expand_model(model)) + + for model in models: + name =3D os.path.join(args.outdir, "x86_{}.xml".format(model["name= "])) + with open(name, "wt") as f: + output_model(f, model) + + +if __name__ =3D=3D "__main__": + main() --=20 2.26.2