From nobody Tue Feb 10 20:47:58 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1686052743; cv=none; d=zohomail.com; s=zohoarc; b=RUr0oH4zEWegQVQtuRSKzXqQSIESy2pnVKYvUzK3wciy5fbBSLB3Yi0lBzlsPLYZhej5uYJHzHmNMFychFCtVMMHnUJOIzq+g3/WcbpE3q2Ny0i4kUOOBX6xzATiqx02ehW2YnIRqBg2KO32+KmSZEnnrR0vf5m77D0kCaEUbEA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1686052743; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Xs7NCRO1NxyKtJ4YtgpLzRcmg5STsaVsT+hkKMwIY5w=; b=VWup2vfKVCQZzezpqROtrKWGJNqJpFGCtjt0I8Eralxsq0Zz+ROSAgZjKH+VgHdboc+Qy1EyPNl3qCbdulpuyURLY2jGFoiePx6Ru2AcQbztg6Gp+R/VKPWyEv/sPsnjqe1YkcVZlL9g1OuaaS0ZNIhMxtGBrTtvqdtZ7NBNj6o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1686052743710896.8446443581432; Tue, 6 Jun 2023 04:59:03 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6VJV-00008b-DP; Tue, 06 Jun 2023 07:57:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6VJT-00007v-Fx for qemu-devel@nongnu.org; Tue, 06 Jun 2023 07:57:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6VJQ-0005j5-V1 for qemu-devel@nongnu.org; Tue, 06 Jun 2023 07:57:23 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-84-QvoJ_1oiNo-3NU1Ih0ikCA-1; Tue, 06 Jun 2023 07:57:18 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ADA0C185A78B for ; Tue, 6 Jun 2023 11:57:18 +0000 (UTC) Received: from localhost (unknown [10.39.208.7]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3096140CFD47; Tue, 6 Jun 2023 11:57:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686052640; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xs7NCRO1NxyKtJ4YtgpLzRcmg5STsaVsT+hkKMwIY5w=; b=KJjaB33novGL2AlnYMF4iaFLAgkNs50Z0F5dGcCyZEEwMsxEVpnZ8Ur5u1iPFMg4Bl/UpG ZPo4pRj1YAvcPPIpq/cV1IxlUar6kueaMvPbxVT2Xv7kXnMLDFJgXH+uE16y5Qxt4qvbFI xMx2lLdXmhUOjnsGIJVWpmN3xNFMcf8= X-MC-Unique: QvoJ_1oiNo-3NU1Ih0ikCA-1 From: marcandre.lureau@redhat.com To: qemu-devel@nongnu.org Cc: Gerd Hoffmann , =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , John Snow , Cleber Rosa Subject: [PATCH 04/21] scripts: add a XML preprocessor script Date: Tue, 6 Jun 2023 15:56:41 +0400 Message-Id: <20230606115658.677673-5-marcandre.lureau@redhat.com> In-Reply-To: <20230606115658.677673-1-marcandre.lureau@redhat.com> References: <20230606115658.677673-1-marcandre.lureau@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=marcandre.lureau@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1686052744406100002 From: Marc-Andr=C3=A9 Lureau gdbus-codegen doesn't support conditions or pre-processing. Rather than duplicating D-Bus interfaces for win32 adaptation, let's have a preprocess step, so we can have platform-specific interfaces. The python script is based on https://github.com/peitaosu/XML-Preprocessor, with bug fixes, some testing and replacing lxml dependency with the built-in xml module. This preprocessing syntax style is not very common, but is similar to the one provided by WiX (https://wixtoolset.org/docs/v3/overview/preprocess= or/) or wixl, that we adopted in QEMU for packaging the guest agent. Signed-off-by: Marc-Andr=C3=A9 Lureau --- MAINTAINERS | 1 + scripts/meson.build | 2 + scripts/xml-preprocess-test.py | 136 +++++++++++++++ scripts/xml-preprocess.py | 293 +++++++++++++++++++++++++++++++++ 4 files changed, 432 insertions(+) create mode 100644 scripts/xml-preprocess-test.py create mode 100755 scripts/xml-preprocess.py diff --git a/MAINTAINERS b/MAINTAINERS index 55668d6336..0119488583 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3194,6 +3194,7 @@ F: docs/interop/dbus* F: docs/sphinx/dbus* F: docs/sphinx/fakedbusdoc.py F: tests/qtest/dbus* +F: scripts/xml-preprocess* =20 Seccomp M: Daniel P. Berrange diff --git a/scripts/meson.build b/scripts/meson.build index 1c89e10a76..532277f5a2 100644 --- a/scripts/meson.build +++ b/scripts/meson.build @@ -1,3 +1,5 @@ if stap.found() install_data('qemu-trace-stap', install_dir: get_option('bindir')) endif + +test('xml-preprocess', files('xml-preprocess-test.py'), suite: ['unit']) diff --git a/scripts/xml-preprocess-test.py b/scripts/xml-preprocess-test.py new file mode 100644 index 0000000000..dd92579969 --- /dev/null +++ b/scripts/xml-preprocess-test.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2023 Red Hat, Inc. +# +# SPDX-License-Identifier: MIT +"""Unit tests for xml-preprocess""" + +import contextlib +import importlib +import os +import platform +import subprocess +import tempfile +import unittest +from io import StringIO + +xmlpp =3D importlib.import_module("xml-preprocess") + + +class TestXmlPreprocess(unittest.TestCase): + """Tests for xml-preprocess.Preprocessor""" + + def test_preprocess_xml(self): + with tempfile.NamedTemporaryFile(mode=3D"w", delete=3DFalse) as te= mp_file: + temp_file.write("") + temp_file_name =3D temp_file.name + result =3D xmlpp.preprocess_xml(temp_file_name) + self.assertEqual(result, "") + os.remove(temp_file_name) + + def test_save_xml(self): + with tempfile.NamedTemporaryFile(mode=3D"w", delete=3DFalse) as te= mp_file: + temp_file_name =3D temp_file.name + xmlpp.save_xml("", temp_file_name) + self.assertTrue(os.path.isfile(temp_file_name)) + os.remove(temp_file_name) + + def test_include(self): + with tempfile.NamedTemporaryFile(mode=3D"w", delete=3DFalse) as in= c_file: + inc_file.write("Content from included file") + inc_file_name =3D inc_file.name + xml_str =3D f"" + expected =3D "Content from included file" + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess(xml_str) + self.assertEqual(result, expected) + os.remove(inc_file_name) + self.assertRaises(FileNotFoundError, xpp.preprocess, xml_str) + + def test_envvar(self): + os.environ["TEST_ENV_VAR"] =3D "TestValue" + xml_str =3D "$(env.TEST_ENV_VAR)" + expected =3D "TestValue" + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess(xml_str) + self.assertEqual(result, expected) + self.assertRaises(KeyError, xpp.preprocess, "$(env.UNKNOWN)") + + def test_sys_var(self): + xml_str =3D "$(sys.ARCH)" + expected =3D f"{platform.architecture()[0]}" + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess(xml_str) + self.assertEqual(result, expected) + self.assertRaises(KeyError, xpp.preprocess, "$(sys.UNKNOWN)") + + def test_cus_var(self): + xml_str =3D "$(var.USER)" + expected =3D "" + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess(xml_str) + self.assertEqual(result, expected) + xml_str =3D "$(var.USER)" + expected =3D "FOO" + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess(xml_str) + self.assertEqual(result, expected) + + def test_error_warning(self): + xml_str =3D "" + expected =3D "" + xpp =3D xmlpp.Preprocessor() + out =3D StringIO() + with contextlib.redirect_stdout(out): + result =3D xpp.preprocess(xml_str) + self.assertEqual(result, expected) + self.assertEqual(out.getvalue(), "[Warning]: test warn\n") + self.assertRaises(RuntimeError, xpp.preprocess, "") + + def test_cmd(self): + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess('= ') + self.assertEqual(result, "hello world") + self.assertRaises( + subprocess.CalledProcessError, + xpp.preprocess, '' + ) + + def test_foreach(self): + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess( + '$(var.x)' + ) + self.assertEqual(result, "abc") + + def test_if_elseif(self): + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess('ok') + self.assertEqual(result, "ok") + result =3D xpp.preprocess('ok') + self.assertEqual(result, "") + result =3D xpp.preprocess('okko<= /root>') + self.assertEqual(result, "ok") + result =3D xpp.preprocess('okko= ') + self.assertEqual(result, "ko") + result =3D xpp.preprocess( + 'okok2ko' + ) + self.assertEqual(result, "ok2") + result =3D xpp.preprocess( + 'okokko' + ) + self.assertEqual(result, "ko") + + def test_ifdef(self): + xpp =3D xmlpp.Preprocessor() + result =3D xpp.preprocess('okko') + self.assertEqual(result, "ko") + result =3D xpp.preprocess( + 'okko' + ) + self.assertEqual(result, "ok") + + +if __name__ =3D=3D "__main__": + unittest.main() diff --git a/scripts/xml-preprocess.py b/scripts/xml-preprocess.py new file mode 100755 index 0000000000..57f1d28912 --- /dev/null +++ b/scripts/xml-preprocess.py @@ -0,0 +1,293 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2017-2019 Tony Su +# Copyright (c) 2023 Red Hat, Inc. +# +# SPDX-License-Identifier: MIT +# +# Adapted from https://github.com/peitaosu/XML-Preprocessor +# +"""This is a XML Preprocessor which can be used to process your XML file b= efore +you use it, to process conditional statements, variables, iteration +statements, error/warning, execute command, etc. + +## XML Schema + +### Include Files +``` + +``` + +### Variables +``` +$(env.EnvironmentVariable) + +$(sys.SystemVariable) + +$(var.CustomVariable) +``` + +### Conditional Statements +``` + + + + + + + + + + + +``` + +### Iteration Statements +``` + + $(var.VARNAME) + +``` + +### Errors and Warnings +``` + + + +``` + +### Commands +``` + +``` +""" + +import os +import platform +import re +import subprocess +import sys +from typing import Optional +from xml.dom import minidom + + +class Preprocessor(): + """This class holds the XML preprocessing state""" + + def __init__(self): + self.sys_vars =3D { + "ARCH": platform.architecture()[0], + "SOURCE": os.path.abspath(__file__), + "CURRENT": os.getcwd(), + } + self.cus_vars =3D {} + + def _pp_include(self, xml_str: str) -> str: + include_regex =3D r"(<\?include([\w\s\\/.:_-]+)\s*\?>)" + matches =3D re.findall(include_regex, xml_str) + for group_inc, group_xml in matches: + inc_file_path =3D group_xml.strip() + with open(inc_file_path, "r", encoding=3D"utf-8") as inc_file: + inc_file_content =3D inc_file.read() + xml_str =3D xml_str.replace(group_inc, inc_file_content) + return xml_str + + def _pp_env_var(self, xml_str: str) -> str: + envvar_regex =3D r"(\$\(env\.(\w+)\))" + matches =3D re.findall(envvar_regex, xml_str) + for group_env, group_var in matches: + xml_str =3D xml_str.replace(group_env, os.environ[group_var]) + return xml_str + + def _pp_sys_var(self, xml_str: str) -> str: + sysvar_regex =3D r"(\$\(sys\.(\w+)\))" + matches =3D re.findall(sysvar_regex, xml_str) + for group_sys, group_var in matches: + xml_str =3D xml_str.replace(group_sys, self.sys_vars[group_var= ]) + return xml_str + + def _pp_cus_var(self, xml_str: str) -> str: + define_regex =3D r"(<\?define\s*(\w+)\s*=3D\s*([\w\s\"]+)\s*\?>)" + matches =3D re.findall(define_regex, xml_str) + for group_def, group_name, group_var in matches: + group_name =3D group_name.strip() + group_var =3D group_var.strip().strip("\"") + self.cus_vars[group_name] =3D group_var + xml_str =3D xml_str.replace(group_def, "") + cusvar_regex =3D r"(\$\(var\.(\w+)\))" + matches =3D re.findall(cusvar_regex, xml_str) + for group_cus, group_var in matches: + xml_str =3D xml_str.replace( + group_cus, + self.cus_vars.get(group_var, "") + ) + return xml_str + + def _pp_foreach(self, xml_str: str) -> str: + foreach_regex =3D r"(<\?foreach\s+(\w+)\s+in\s+([\w;]+)\s*\?>(.*)<= \?endforeach\?>)" + matches =3D re.findall(foreach_regex, xml_str) + for group_for, group_name, group_vars, group_text in matches: + group_texts =3D "" + for var in group_vars.split(";"): + self.cus_vars[group_name] =3D var + group_texts +=3D self._pp_cus_var(group_text) + xml_str =3D xml_str.replace(group_for, group_texts) + return xml_str + + def _pp_error_warning(self, xml_str: str) -> str: + error_regex =3D r"<\?error\s*\"([^\"]+)\"\s*\?>" + matches =3D re.findall(error_regex, xml_str) + for group_var in matches: + raise RuntimeError("[Error]: " + group_var) + warning_regex =3D r"(<\?warning\s*\"([^\"]+)\"\s*\?>)" + matches =3D re.findall(warning_regex, xml_str) + for group_wrn, group_var in matches: + print("[Warning]: " + group_var) + xml_str =3D xml_str.replace(group_wrn, "") + return xml_str + + def _pp_if_eval(self, xml_str: str) -> str: + ifelif_regex =3D ( + r"(<\?(if|elseif)\s*([^\"\s=3D<>!]+)\s*([!=3D<>]+)\s*\"*([^\"= =3D<>!]+)\"*\s*\?>)" + ) + matches =3D re.findall(ifelif_regex, xml_str) + for ifelif, tag, left, operator, right in matches: + if "<" in operator or ">" in operator: + result =3D eval(f"{left} {operator} {right}") + else: + result =3D eval(f'"{left}" {operator} "{right}"') + xml_str =3D xml_str.replace(ifelif, f"") + return xml_str + + def _pp_ifdef_ifndef(self, xml_str: str) -> str: + ifndef_regex =3D r"(<\?(ifdef|ifndef)\s*([\w]+)\s*\?>)" + matches =3D re.findall(ifndef_regex, xml_str) + for group_ifndef, group_tag, group_var in matches: + if group_tag =3D=3D "ifdef": + result =3D group_var in self.cus_vars + else: + result =3D group_var not in self.cus_vars + xml_str =3D xml_str.replace(group_ifndef, f"") + return xml_str + + def _pp_if_elseif(self, xml_str: str) -> str: + if_elif_else_regex =3D ( + r"(<\?if\s(True|False)\?>" + r"(.*?)" + r"<\?elseif\s(True|False)\?>" + r"(.*?)" + r"<\?else\?>" + r"(.*?)" + r"<\?endif\?>)" + ) + if_else_regex =3D ( + r"(<\?if\s(True|False)\?>" + r"(.*?)" + r"<\?else\?>" + r"(.*?)" + r"<\?endif\?>)" + ) + if_regex =3D r"(<\?if\s(True|False)\?>(.*?)<\?endif\?>)" + matches =3D re.findall(if_elif_else_regex, xml_str, re.DOTALL) + for (group_full, group_if, group_if_elif, group_elif, + group_elif_else, group_else) in matches: + result =3D "" + if group_if =3D=3D "True": + result =3D group_if_elif + elif group_elif =3D=3D "True": + result =3D group_elif_else + else: + result =3D group_else + xml_str =3D xml_str.replace(group_full, result) + matches =3D re.findall(if_else_regex, xml_str, re.DOTALL) + for group_full, group_if, group_if_else, group_else in matches: + result =3D "" + if group_if =3D=3D "True": + result =3D group_if_else + else: + result =3D group_else + xml_str =3D xml_str.replace(group_full, result) + matches =3D re.findall(if_regex, xml_str, re.DOTALL) + for group_full, group_if, group_text in matches: + result =3D "" + if group_if =3D=3D "True": + result =3D group_text + xml_str =3D xml_str.replace(group_full, result) + return xml_str + + def _pp_command(self, xml_str: str) -> str: + cmd_regex =3D r"(<\?cmd\s*\"([^\"]+)\"\s*\?>)" + matches =3D re.findall(cmd_regex, xml_str) + for group_cmd, group_exec in matches: + output =3D subprocess.check_output( + group_exec, shell=3DTrue, + text=3DTrue, stderr=3Dsubprocess.STDOUT + ) + xml_str =3D xml_str.replace(group_cmd, output) + return xml_str + + def _pp_blanks(self, xml_str: str) -> str: + right_blank_regex =3D r">[\n\s\t\r]*" + left_blank_regex =3D r"[\n\s\t\r]*<" + xml_str =3D re.sub(right_blank_regex, ">", xml_str) + xml_str =3D re.sub(left_blank_regex, "<", xml_str) + return xml_str + + def preprocess(self, xml_str: str) -> str: + fns =3D [ + self._pp_blanks, + self._pp_include, + self._pp_foreach, + self._pp_env_var, + self._pp_sys_var, + self._pp_cus_var, + self._pp_if_eval, + self._pp_ifdef_ifndef, + self._pp_if_elseif, + self._pp_command, + self._pp_error_warning, + ] + + while True: + changed =3D False + for func in fns: + out_xml =3D func(xml_str) + if not changed and out_xml !=3D xml_str: + changed =3D True + xml_str =3D out_xml + if not changed: + break + + return xml_str + + +def preprocess_xml(path: str) -> str: + with open(path, "r", encoding=3D"utf-8") as original_file: + input_xml =3D original_file.read() + + proc =3D Preprocessor() + return proc.preprocess(input_xml) + + +def save_xml(xml_str: str, path: Optional[str]): + xml =3D minidom.parseString(xml_str) + with open(path, "w", encoding=3D"utf-8") if path else sys.stdout as ou= tput_file: + output_file.write(xml.toprettyxml()) + + +def main(): + if len(sys.argv) < 2: + print("Usage: xml-preprocessor input.xml [output.xml]") + sys.exit(1) + + output_file =3D None + if len(sys.argv) =3D=3D 3: + output_file =3D sys.argv[2] + + input_file =3D sys.argv[1] + output_xml =3D preprocess_xml(input_file) + save_xml(output_xml, output_file) + + +if __name__ =3D=3D "__main__": + main() --=20 2.40.1