From nobody Mon Feb 9 13:21:21 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) client-ip=216.205.24.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1602771513; cv=none; d=zohomail.com; s=zohoarc; b=fk+heeFMnC2fPPKYdksFOs4YQ19EAkISbuDoxzP6oKojIIZ9fJOMIFdbT6bGN65Qi9g8jLtjgFpEtX8lwX4PXTwKsFcpWXPgUrqAEWuwb9iUdQNnextkTSjrr6bOzHg3Ln8cZ3e5dI2/nH8qxTChu220oUJfkrZ87B/I6kqJckM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602771513; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=f5WmJF6rrejNIjcPzZv7s0CRi94jdhJtgv5ZNVxqYsY=; b=GzWmiKLhiiCbxG6cKOuZxtOHgAM2jC7JaDbinn/5NKMWILaszT5NBBJNpPrmHpLUjy9lb4G6pJxL9m4Qqh7bgkI2wJSvY7Uh2F13FXGCy3LtXqnLimQsXqV1zi0ZePP3p1wblJbI5IF/EFjh0puzVyXCNYCknqLgRjhnZcE6sXs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mx.zohomail.com with SMTPS id 160277151343457.939699541573304; Thu, 15 Oct 2020 07:18:33 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-243-gfJ1dEJhOni3emm-GICaiA-1; Thu, 15 Oct 2020 10:18:27 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AF223640A1; Thu, 15 Oct 2020 14:18:19 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8D8F75D9DD; Thu, 15 Oct 2020 14:18:19 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 32247181A87F; Thu, 15 Oct 2020 14:18:19 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 09FEIDlZ006957 for ; Thu, 15 Oct 2020 10:18:13 -0400 Received: by smtp.corp.redhat.com (Postfix) id 57B616EF65; Thu, 15 Oct 2020 14:18:13 +0000 (UTC) Received: from work.redhat.com (ovpn-112-170.ams2.redhat.com [10.36.112.170]) by smtp.corp.redhat.com (Postfix) with ESMTP id 96BC36EF67; Thu, 15 Oct 2020 14:18:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1602771511; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=f5WmJF6rrejNIjcPzZv7s0CRi94jdhJtgv5ZNVxqYsY=; b=Z4DiZvPZ85kA8Lx842dVWoOmeW+dAz2esr6477/0g6/QR7d0AJTr/Oz0uXNVoeaVUmDOx5 Az2jY1HnOec7ASBsMQ6a+bKT36eaHUoQPE4AGghiANCEELSmU7xN84Y0zgHEWwrYK0Xad8 JJcb5oWBBuXV29h5s9FT7rLES925Rvk= X-MC-Unique: gfJ1dEJhOni3emm-GICaiA-1 From: Tim Wiederhake To: libvir-list@redhat.com Subject: [libvirt PATCH 2/3] cpu_map: Add script to sync from QEMU i386 cpu models Date: Thu, 15 Oct 2020 16:18:04 +0200 Message-Id: <20201015141805.143762-3-twiederh@redhat.com> In-Reply-To: <20201015141805.143762-1-twiederh@redhat.com> References: <20201015141805.143762-1-twiederh@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Cc: Tim Wiederhake X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This script is intended to help in synchronizing i386 QEMU cpu model definitions with libvirt. As the QEMU cpu model definitions are post processed by QEMU and not meant to be consumed by third parties directly, parsing this information is imperfect. Additionally, the libvirt models contain information that cannot be generated from the QEMU data, preventing fully automated usage. The output should nevertheless be helpful for a human in determining potentially interesting changes. Signed-off-by: Tim Wiederhake --- src/cpu_map/sync_qemu_i386.py | 361 ++++++++++++++++++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100755 src/cpu_map/sync_qemu_i386.py diff --git a/src/cpu_map/sync_qemu_i386.py b/src/cpu_map/sync_qemu_i386.py new file mode 100755 index 0000000000..cdc5426f3d --- /dev/null +++ b/src/cpu_map/sync_qemu_i386.py @@ -0,0 +1,361 @@ +#!/usr/bin/env python3 + +import argparse +import copy +import json +import lark +import os +import re + + +T =3D { + # translating qemu -> libvirt cpu vendor names + "CPUID_VENDOR_AMD": "AMD", + "CPUID_VENDOR_INTEL": "Intel", + "CPUID_VENDOR_HYGON": "Hygon", + + # translating qemu -> libvirt cpu feature names + "CPUID_6_EAX_ARAT": "arat", + "CPUID_7_0_EBX_ADX": "adx", + "CPUID_7_0_EBX_AVX2": "avx2", + "CPUID_7_0_EBX_AVX512BW": "avx512bw", + "CPUID_7_0_EBX_AVX512CD": "avx512cd", + "CPUID_7_0_EBX_AVX512DQ": "avx512dq", + "CPUID_7_0_EBX_AVX512ER": "avx512er", + "CPUID_7_0_EBX_AVX512F": "avx512f", + "CPUID_7_0_EBX_AVX512PF": "avx512pf", + "CPUID_7_0_EBX_AVX512VL": "avx512vl", + "CPUID_7_0_EBX_BMI1": "bmi1", + "CPUID_7_0_EBX_BMI2": "bmi2", + "CPUID_7_0_EBX_CLFLUSHOPT": "clflushopt", + "CPUID_7_0_EBX_CLWB": "clwb", + "CPUID_7_0_EBX_ERMS": "erms", + "CPUID_7_0_EBX_FSGSBASE": "fsgsbase", + "CPUID_7_0_EBX_HLE": "hle", + "CPUID_7_0_EBX_INVPCID": "invpcid", + "CPUID_7_0_EBX_MPX": "mpx", + "CPUID_7_0_EBX_RDSEED": "rdseed", + "CPUID_7_0_EBX_RTM": "rtm", + "CPUID_7_0_EBX_SHA_NI": "sha-ni", + "CPUID_7_0_EBX_SMAP": "smap", + "CPUID_7_0_EBX_SMEP": "smep", + "CPUID_7_0_ECX_AVX512BITALG": "avx512bitalg", + "CPUID_7_0_ECX_AVX512_VBMI2": "avx512vbmi2", + "CPUID_7_0_ECX_AVX512_VBMI": "avx512vbmi", + "CPUID_7_0_ECX_AVX512VNNI": "avx512vnni", + "CPUID_7_0_ECX_AVX512_VPOPCNTDQ": "avx512-vpopcntdq", + "CPUID_7_0_ECX_CLDEMOTE": "cldemote", + "CPUID_7_0_ECX_GFNI": "gfni", + "CPUID_7_0_ECX_LA57": "la57", + "CPUID_7_0_ECX_MOVDIR64B": "movdir64b", + "CPUID_7_0_ECX_MOVDIRI": "movdiri", + "CPUID_7_0_ECX_PKU": "pku", + "CPUID_7_0_ECX_RDPID": "rdpid", + "CPUID_7_0_ECX_UMIP": "umip", + "CPUID_7_0_ECX_VAES": "vaes", + "CPUID_7_0_ECX_VPCLMULQDQ": "vpclmulqdq", + "CPUID_7_0_EDX_ARCH_CAPABILITIES": "arch-capabilities", + "CPUID_7_0_EDX_AVX512_4FMAPS": "avx512-4fmaps", + "CPUID_7_0_EDX_AVX512_4VNNIW": "avx512-4vnniw", + "CPUID_7_0_EDX_CORE_CAPABILITY": "core-capability", + "CPUID_7_0_EDX_SPEC_CTRL": "spec-ctrl", + "CPUID_7_0_EDX_SPEC_CTRL_SSBD": "ssbd", + "CPUID_7_0_EDX_STIBP": "stibp", + "CPUID_7_1_EAX_AVX512_BF16": "avx512-bf16", + "CPUID_8000_0008_EBX_CLZERO": "clzero", + "CPUID_8000_0008_EBX_IBPB": "ibpb", + "CPUID_8000_0008_EBX_STIBP": "amd-stibp", + "CPUID_8000_0008_EBX_WBNOINVD": "wbnoinvd", + "CPUID_8000_0008_EBX_XSAVEERPTR": "xsaveerptr", + "CPUID_ACPI": "acpi", + "CPUID_APIC": "apic", + "CPUID_CLFLUSH": "clflush", + "CPUID_CMOV": "cmov", + "CPUID_CX8": "cx8", + "CPUID_DE": "de", + "CPUID_EXT2_3DNOW": "3dnow", + "CPUID_EXT2_3DNOWEXT": "3dnowext", + "CPUID_EXT2_FFXSR": "fxsr_opt", + "CPUID_EXT2_LM": "lm", + "CPUID_EXT2_MMXEXT": "mmxext", + "CPUID_EXT2_NX": "nx", + "CPUID_EXT2_PDPE1GB": "pdpe1gb", + "CPUID_EXT2_RDTSCP": "rdtscp", + "CPUID_EXT2_SYSCALL": "syscall", + "CPUID_EXT3_3DNOWPREFETCH": "3dnowprefetch", + "CPUID_EXT3_ABM": "abm", + "CPUID_EXT3_CR8LEG": "cr8legacy", + "CPUID_EXT3_FMA4": "fma4", + "CPUID_EXT3_LAHF_LM": "lahf_lm", + "CPUID_EXT3_MISALIGNSSE": "misalignsse", + "CPUID_EXT3_OSVW": "osvw", + "CPUID_EXT3_PERFCORE": "perfctr_core", + "CPUID_EXT3_SSE4A": "sse4a", + "CPUID_EXT3_SVM": "svm", + "CPUID_EXT3_TBM": "tbm", + "CPUID_EXT3_XOP": "xop", + "CPUID_EXT_AES": "aes", + "CPUID_EXT_AVX": "avx", + "CPUID_EXT_CX16": "cx16", + "CPUID_EXT_F16C": "f16c", + "CPUID_EXT_FMA": "fma", + "CPUID_EXT_MOVBE": "movbe", + "CPUID_EXT_PCID": "pcid", + "CPUID_EXT_PCLMULQDQ": "pclmuldq", + "CPUID_EXT_POPCNT": "popcnt", + "CPUID_EXT_RDRAND": "rdrand", + "CPUID_EXT_SSE3": "pni", + "CPUID_EXT_SSE41": "sse4.1", + "CPUID_EXT_SSE42": "sse4.2", + "CPUID_EXT_SSSE3": "ssse3", + "CPUID_EXT_TSC_DEADLINE_TIMER": "tsc-deadline", + "CPUID_EXT_X2APIC": "x2apic", + "CPUID_EXT_XSAVE": "xsave", + "CPUID_FP87": "fpu", + "CPUID_FXSR": "fxsr", + "CPUID_MCA": "mca", + "CPUID_MCE": "mce", + "CPUID_MMX": "mmx", + "CPUID_MSR": "msr", + "CPUID_MTRR": "mtrr", + "CPUID_PAE": "pae", + "CPUID_PAT": "pat", + "CPUID_PGE": "pge", + "CPUID_PSE36": "pse36", + "CPUID_PSE": "pse", + "CPUID_SEP": "sep", + "CPUID_SSE2": "sse2", + "CPUID_SSE": "sse", + "CPUID_SS": "ss", + "CPUID_SVM_NPT": "npt", + "CPUID_SVM_NRIPSAVE": "nrip-save", + "CPUID_TSC": "tsc", + "CPUID_VME": "vme", + "CPUID_XSAVE_XGETBV1": "xgetbv1", + "CPUID_XSAVE_XSAVEC": "xsavec", + "CPUID_XSAVE_XSAVEOPT": "xsaveopt", + "CPUID_XSAVE_XSAVES": "xsaves", + "MSR_ARCH_CAP_IBRS_ALL": "ibrs-all", + "MSR_ARCH_CAP_MDS_NO": "mds-no", + "MSR_ARCH_CAP_PSCHANGE_MC_NO": "pschange-mc-no", + "MSR_ARCH_CAP_RDCL_NO": "rdctl-no", + "MSR_ARCH_CAP_SKIP_L1DFL_VMENTRY": "skip-l1dfl-vmentry", + "MSR_ARCH_CAP_TAA_NO": "taa-no", + "MSR_CORE_CAP_SPLIT_LOCK_DETECT": "split-lock-detect", + + # always disabled features + "CPUID_EXT_MONITOR": None, + "0": None, + + # set to "no auto enable" by qemu + "CPUID_EXT3_TOPOEXT": None, + "MSR_VMX_BASIC_DUAL_MONITOR": None, +} + + +def readline_cont(f): + """Read one logical line from a file `f` i.e. continues lines that end= in + a backslash.""" + + line =3D f.readline() + while line.endswith("\\\n"): + line =3D line[:-2] + " " + f.readline() + return line + + +def read_builtin_x86_defs(filename): + """Extract content between begin_mark and end_mark from file `filename= ` as + string, while expanding shorthand macros like "I486_FEATURES".""" + + begin_mark =3D "static X86CPUDefinition builtin_x86_defs[] =3D {\n" + end_mark =3D "};\n" + shorthand =3D re.compile("^#define ([A-Z0-9_]+_FEATURES) (.*)$") + lines =3D list() + shorthands =3D dict() + + with open(filename, "rt") as f: + while (line :=3D readline_cont(f)) !=3D begin_mark: + if not line: + raise RuntimeError("begin mark not found") + if match :=3D shorthand.match(line): + # TCG definitions are irrelevant for cpu models + newk =3D match.group(1) + if newk.startswith("TCG_"): + continue + + # remove comments, whitespace and bit operators, effective= ly + # turning the bitfield into a list + newv =3D re.sub("([()|\t\n])|(/\*.*?\*/)", " ", match.grou= p(2)) + + # resolve recursive shorthands + for k, v in shorthands.items(): + newv =3D newv.replace(k, v) + + shorthands[newk] =3D newv + + while (line :=3D readline_cont(f)) !=3D end_mark: + if not line: + raise RuntimeError("end marker not found") + + # apply shorthands + for k, v in shorthands.items(): + line =3D line.replace(k, v) + lines.append(line) + + return "".join(lines) + + +def transform(item): + """Recursively transform a Lark syntax tree into python native objects= .""" + + if isinstance(item, lark.lexer.Token): + return str(item) + + if item.data =3D=3D "list": + l =3D list() + for child in item.children: + value =3D transform(child) + if value is None: + continue + l.append(value) + return l + + if item.data =3D=3D "map": + d =3D dict() + for child in item.children: + if len(child.children) !=3D 2: + raise RuntimeError("map entry with more than 2 elements") + key =3D transform(child.children[0]) + value =3D transform(child.children[1]) + if key is None: + raise RuntimeError("map entry with 'None' key") + if value is None: + continue + d[key] =3D value + return d + + if item.data =3D=3D "text": + l =3D list() + for child in item.children: + value =3D transform(child) + if value is None: + continue + l.append(value) + return " ".join(l) + + if item.data =3D=3D "value": + if item.children: + raise RuntimeError("empty list is not empty") + return None + + raise RuntimeError("unexpected item type") + + +def expand_model(model): + """Expand a qemu cpu model description that has its feature split up i= nto + different fields and may have differing versions into several libvirt- + friendly cpu models.""" + + result =3D { + "name": model.pop(".name"), + "vendor": T[model.pop(".vendor")], + "features": set(), + "extra": dict()} + + if ".family" in model and ".model" in model: + result["family"] =3D model.pop(".family") + result["model"] =3D model.pop(".model") + + for k in [k for k in model if k.startswith(".features")]: + v =3D model.pop(k) + for feature in v.split(): + if feature.startswith("VMX_") or feature.startswith("MSR_VMX_"= ): + continue + translated =3D T.get(feature, feature) + if translated: + result["features"].add(translated) + + versions =3D model.pop(".versions", []) + for k, v in model.items(): + result["extra"]["model" + k] =3D v + yield result + + for version in versions: + result =3D copy.deepcopy(result) + result["name"] =3D version.pop(".alias", result["name"]) + + props =3D version.pop(".props", dict()) + for k, v in props: + if v =3D=3D "on": + result["features"].add(k) + elif v =3D=3D "off" and k in result["features"]: + result["features"].remove(k) + else: + result["extra"]["property." + k] =3D v + + for k, v in version.items(): + result["extra"]["version" + k] =3D v + + yield result + + +def output_model(f, model): + if model["extra"]: + f.write("\n") + + f.write("\n") + f.write(" \n".format(model["name"])) + f.write(" \n") + f.write(" \n".format( + model["family"], model["model"])) + f.write(" \n".format(model["vendor"])) + for feature in sorted(model["features"]): + f.write(" \n".format(feature)) + f.write(" \n") + f.write("\n") + + +def main(): + parser =3D argparse.ArgumentParser( + description =3D "Synchronize x86 cpu models from QEMU i386 target.= ") + parser.add_argument("cpufile", + help=3D"Path to 'target/i386/cpu.c' file in the QEMU repository", + type=3Dos.path.realpath) + parser.add_argument("outdir", + help=3D"Path to 'src/cpu_map' directory in the libvirt repository", + type=3Dos.path.realpath) + + args =3D parser.parse_args() + + builtin_x86_defs =3D read_builtin_x86_defs(args.cpufile) + + ast =3D lark.Lark(r""" + list: value ( "," value )* ","? + map: keyvalue ( "," keyvalue )* ","? + keyvalue: IDENTIFIER "=3D" value + ?value: text | "{" "}" | "{" list "}" | "{" map "}" + text: (IDENTIFIER | "\"" (/[^"]+/)? "\"")+ + IDENTIFIER: /[\[\]\._&a-zA-Z0-9]/+ + %ignore (" " | "\r" | "\n" | "\t" | "|" )+ + %ignore "(" ( "X86CPUVersionDefinition" | "PropValue" ) "[])" + %ignore "//" /.*?/ "\n" + %ignore "/*" /(.|\n)*?/ "*/" + """, start=3D"list").parse(builtin_x86_defs) + + models_json =3D transform(ast) + + models =3D list() + for model in models_json: + models.extend(expand_model(model)) + + for model in models: + name =3D os.path.join(args.outdir, "x86_{}.xml".format(model["name= "])) + with open(name, "wt") as f: + output_model(f, model) + + +if __name__ =3D=3D "__main__": + main() --=20 2.26.2