From nobody Tue Feb 10 02:00:39 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1686151138; cv=none; d=zohomail.com; s=zohoarc; b=Th9wW6UNL8aZjp+pvyRTosnVCGI2tAbWvlPBlISQ+YkavFa6+cAnmi5DctffRS3rQ7z1J9HqbJHRKobGi5+E7PVYGk5dBbfmpy12iEIO2xKcGyeX7BU1wN5ArFvUX0Q1bPSuqPkUBpoAxYqjQp5/ZJ7Qr9aKZ0A7QuJerOgYIrA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1686151138; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=uCvUSXHQKvX0jdw+5XBdhLoCUhMUlRKK0HNJdbWfD5k=; b=BQ3OyFBY6GHmp2I4/Fu/Laj5hTRWms1HlTdOEcRgK8Y2m0hqHmpLPcXqZgjUqBL2T7slP/xIUU8w8nvnOURWlIaLrCSHgRHmp4LCAg27X9gaNegtT8ItfbBOheCYsVdQmlm9wnVnYnKtt3Nypj5x1oPBz8LxHc6QFqOSGPmDxws= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1686151138605543.5735382025592; Wed, 7 Jun 2023 08:18:58 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-500-f8MrE6jJNeqPetDjT87gLA-1; Wed, 07 Jun 2023 11:18:53 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8EDAD1C068F9; Wed, 7 Jun 2023 15:18:32 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id B4B7340C1438; Wed, 7 Jun 2023 15:18:31 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 87CED19452C7; Wed, 7 Jun 2023 15:18:20 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id D02A419465BA for ; Wed, 7 Jun 2023 15:08:47 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 9517C48205E; Wed, 7 Jun 2023 15:08:47 +0000 (UTC) Received: from speedmetal.lan (unknown [10.45.242.14]) by smtp.corp.redhat.com (Postfix) with ESMTP id EAC21492B00 for ; Wed, 7 Jun 2023 15:08:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686151137; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=uCvUSXHQKvX0jdw+5XBdhLoCUhMUlRKK0HNJdbWfD5k=; b=g/1R5RGhNz3QmmrIW3qiz0gEZUbKO2ojgJqoBKU5NhKntHVfX012R1MCDbAUfYswBpRhjy RYm2yajFygJat2jgWAShr47vzLPknDKWZ6D8CrqhmDDLWgAfG7HlLBtRbFo9+V+UWa6dAY DKs24dG1T6zKT2uS1++UbY49JVNO+gQ= X-MC-Unique: f8MrE6jJNeqPetDjT87gLA-1 X-Original-To: libvir-list@listman.corp.redhat.com From: Peter Krempa To: libvir-list@redhat.com Subject: [PATCH] scripts: QMP schema query string helper script Date: Wed, 7 Jun 2023 17:08:45 +0200 Message-Id: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1686151140493100001 Content-Type: text/plain; charset="utf-8" The script generates all query strings in same format as used for querying qemu capabilities for a given .replies file. The output can be used to either aid in creation of the query strings as well as to generate diff between the schema, which is useful when adding a new capability dump. The script also validates that all of the schema is supported by our tools so that we can always adapt. The output looks like: $ ./scripts/qapi-schema-diff-gen.py tests/qemucapabilitiesdata/caps_8.0.0_= x86_64.replies [...] query-yank query-yank/ret-type/type query-yank/ret-type/type/^block-node query-yank/ret-type/type/^chardev query-yank/ret-type/type/^migration query-yank/ret-type/+block-node query-yank/ret-type/+block-node/node-name query-yank/ret-type/+block-node/node-name/!str query-yank/ret-type/+chardev query-yank/ret-type/+chardev/id query-yank/ret-type/+chardev/id/!str query-yank/ret-type/+migration [...] Signed-off-by: Peter Krempa Reviewed-by: Jonathon Jongsma --- scripts/meson.build | 1 + scripts/qapi-schema-diff-gen.py | 313 ++++++++++++++++++++++++++++++++ tests/meson.build | 12 ++ 3 files changed, 326 insertions(+) create mode 100755 scripts/qapi-schema-diff-gen.py diff --git a/scripts/meson.build b/scripts/meson.build index 05b71184f1..c216c7e1ff 100644 --- a/scripts/meson.build +++ b/scripts/meson.build @@ -30,6 +30,7 @@ scripts =3D [ 'meson-timestamp.py', 'mock-noinline.py', 'prohibit-duplicate-header.py', + 'qapi-schema-diff-gen.py', ] foreach name : scripts diff --git a/scripts/qapi-schema-diff-gen.py b/scripts/qapi-schema-diff-gen= .py new file mode 100755 index 0000000000..bde130e33c --- /dev/null +++ b/scripts/qapi-schema-diff-gen.py @@ -0,0 +1,313 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# A tool to help with creating query strings for querying the QMP schema (= as +# returned by 'query-qmp-schema' QMP command). In default mode it generate= s all +# the possible query strings for a QMP schema in the ".replies" format as +# generated by 'tests/qemucapsprobe.c'. This can be either used by users to +# find the desired schema query string or to see what changed between two +# versions. +# +# In the '--validate' mode the script doesn't output the schema query stri= ngs. +# This invokes just the validator that everything in the schema is support= ed by +# the tool. +# +# Note: Any change to the 'validate_schema' function below to make it acce= pt +# new schema components most likely requires change to either +# 'src/qemu/qemu_qapi.c' or 'tests/testutilsqemuschema.c' to accept the new +# schema components. + +from pathlib import Path +import argparse +import json +import sys + + +# Finds the apropriate call to 'query-qmp-schema' in the '.replies' file a= nd +# returns the JSON blob following the command invocation. +def load_schema_json_list(filename): + found =3D False + + with open(filename, "r") as fh: + jsonstr =3D '' + for line in fh: + jsonstr +=3D line + + if line !=3D '}\n': + continue + + if found: + return json.loads(jsonstr)["return"] + + cmdobj =3D json.loads(jsonstr) + jsonstr =3D "" + + if isinstance(cmdobj, dict) and cmdobj.get('execute', '') =3D= =3D 'query-qmp-schema': + found =3D True + + +# Validates that 'entry' (an member of the QMP schema): +# - checks that it's a Dict (imported from a JSON object) +# - checks that all 'mandatory' fields are present and their types match +# - checks the types of all 'optional' fields +# - checks that no unknown fields are present +def check_keys(entry, mandatory, optional): + keys =3D set(entry.keys()) + + for k, t in mandatory: + try: + keys.remove(k) + except KeyError: + raise Exception("missing mandatory key '%s' in schema '%s'" % = (k, entry)) + + if not isinstance(entry[k], t): + raise Exception("key '%s' is not of the expected type '%s' in = schema '%s'" % (k, t, entry)) + + for k, t in optional: + if k in keys: + keys.discard(k) + + if t is not None: + if not isinstance(entry[k], t): + raise Exception("key '%s' is not of the expected type = '%s' in schema '%s'" % (k, t, entry)) + + if len(keys) > 0: + raise Exception("unhandled keys '%s' in schema '%s'" % (','.join(l= ist(keys)), entry)) + + +# Validates the optional 'features' and that they consist only of strings +def check_features_list(entry): + for f in entry.get('features', []): + if not isinstance(f, str): + raise Exception("broken 'features' list in schema entry '%s'" = % entry) + + +# Validate that the passed schema has only supported members. This is usef= ul to +# stay up to date with any changes to the schema. +def validate_schema(schemalist): + for entry in schemalist: + if not isinstance(entry, dict): + raise Exception("schema entry '%s' is not a JSON Object (dict)= " % (entry)) + + match entry.get('meta-type', None): + case 'command': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str), + ('arg-type', str), + ('ret-type', str)], + optional=3D[('features', list), + ('allow-oob', bool)]) + + check_features_list(entry) + + case 'event': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str), + ('arg-type', str)], + optional=3D[('features', list)]) + + check_features_list(entry) + + case 'object': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str), + ('members', list)], + optional=3D[('tag', str), + ('variants', list), + ('features', list)]) + + check_features_list(entry) + + for m in entry.get('members', []): + check_keys(m, + mandatory=3D[('name', str), + ('type', str)], + optional=3D[('default', None), + ('features', list)]) + check_features_list(m) + + for m in entry.get('variants', []): + check_keys(m, + mandatory=3D[('case', str), + ('type', str)], + optional=3D[]) + + case 'array': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str), + ('element-type', str)], + optional=3D[]) + + case 'enum': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str)], + optional=3D[('members', list), + ('values', list)]) + + for m in entry.get('members', []): + check_keys(m, + mandatory=3D[('name', str)], + optional=3D[('features', list)]) + check_features_list(m) + + case 'alternate': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str), + ('members', list)], + optional=3D[]) + + for m in entry.get('members', []): + check_keys(m, + mandatory=3D[('type', str)], + optional=3D[]) + case 'builtin': + check_keys(entry, + mandatory=3D[('name', str), + ('meta-type', str), + ('json-type', str)], + optional=3D[]) + + case _: + raise Exception("unknown or missing 'meta-type' in schema = entry '%s'" % entry) + + +# Convert a list of QMP schema entries into a dict organized via 'name' me= mber +def load_schema_json_dict(schemalist): + schemadict =3D {} + + for memb in schemalist: + schemadict[memb['name']] =3D memb + + return schemadict + + +# loads and validates the QMP schema from the .replies file 'filename' +def load_schema(filename): + schemalist =3D load_schema_json_list(filename) + + if not schemalist: + raise Exception("QMP schema not found in '%s'" % (filename)) + + validate_schema(schemalist) + + return load_schema_json_dict(schemalist) + + +# Recursively traverse the schema and print out the schema query strings f= or +# the corresponding entries. In certain cases the schema references itself, +# which is handled by passing a 'trace' list which contains the current pa= th +def iterate_schema(name, cur, trace, schema): + obj =3D schema[name] + + if name in trace: + print('%s (recursion)' % cur) + return + + trace =3D trace + [name] + + match obj['meta-type']: + case 'command' | 'event': + arguments =3D obj.get('arg-type', None) + returns =3D obj.get('ret-type', None) + + print(name) + + for f in obj.get('features', []): + print('%s/$%s' % (cur, f)) + + if arguments: + iterate_schema(arguments, cur + '/arg-type', trace, schema) + + if returns: + iterate_schema(returns, cur + '/ret-type', trace, schema) + + case 'object': + members =3D sorted(obj.get('members', []), key=3Dlambda d: d['= name']) + variants =3D sorted(obj.get('variants', []), key=3Dlambda d: d= ['case']) + + for f in obj.get('features', []): + print('%s/$%s' % (cur, f)) + + for memb in members: + membpath =3D "%s/%s" % (cur, memb['name']) + print(membpath) + + for f in memb.get('features', []): + print('%s/$%s' % (membpath, f)) + + iterate_schema(memb['type'], membpath, trace, schema) + + for var in variants: + varpath =3D "%s/+%s" % (cur, var['case']) + print(varpath) + iterate_schema(var['type'], varpath, trace, schema) + + case 'enum': + members =3D sorted(obj.get('members', []), key=3Dlambda d: d['= name']) + + for m in members: + print('%s/^%s' % (cur, m['name'])) + + for f in m.get('features', []): + print('%s/^%s/$%s' % (cur, m['name'], f)) + + case 'array': + iterate_schema(obj['element-type'], cur, trace, schema) + + case 'builtin': + print('%s/!%s' % (cur, name)) + + case 'alternate': + for var in obj['members']: + iterate_schema(var['type'], cur, trace, schema) + + case _: + raise Exception("unhandled 'meta-type' '%s'" % obj.get('meta-t= ype', '')) + + +def process_one_schema(schemafile, validate_only): + try: + schema =3D load_schema(schemafile) + except Exception as e: + raise Exception("Failed to load schema '%s': %s" % (schemafile, e)) + + if validate_only: + return + + toplevel =3D [] + + for k, v in schema.items(): + if v['meta-type'] =3D=3D 'command' or v['meta-type'] =3D=3D 'event= ': + toplevel.append(k) + + toplevel.sort() + + for c in toplevel: + iterate_schema(c, c, [], schema) + + +parser =3D argparse.ArgumentParser(description=3D'A tool to generate QMP s= chema query strins and validator of schema coverage') +parser.add_argument('--validate', action=3D"store_true", help=3D'only load= the schema and validate it') +parser.add_argument('--schemadir', default=3D'', + help=3D'directory containing .replies files') +parser.add_argument('schema', nargs=3D'?', help=3D'path to .replies file t= o use') +args =3D parser.parse_args() + +if not args.schema and not args.schemadir: + parser.print_help() + sys.exit(1) + +if args.schema: + process_one_schema(args.schema, args.validate) +else: + files =3D Path(args.schemadir).glob('*.replies') + + for file in files: + process_one_schema(str(file), args.validate) diff --git a/tests/meson.build b/tests/meson.build index 0082446029..25e7ccd312 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -598,6 +598,18 @@ foreach data : tests test(data['name'], test_bin, env: tests_env, timeout: timeout, depends: = tests_deps) endforeach +test( + 'qapi-schema-check', + python3_prog, + args: [ + qapi_schema_diff_gen_prog.full_path(), + '--validate', + '--schemadir', + meson.project_source_root() / 'tests' / 'qemucapabilitiesdata' + ], + env: runutf8, +) + # helpers: # each entry is a dictionary with following items: --=20 2.40.1