From nobody Tue May 14 09:22:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1680037939; cv=none; d=zohomail.com; s=zohoarc; b=MgTS73Y+1pD9GTx57n9rlpJo5IXwZJsqsaV6YEpzzEqm9GUCWFZ+Niu7Z3XicRXTkoC+SHfkIg/37FU8Nnqpwol2I7Erks0MTP2JHYlYB6/Hu/JaEDpKBSQE5++vFbORWHZFIOPk0kVcVvzP3f6I966IQwmOfUiAM9Sf+BoN4oI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1680037939; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=+gcVU8asKsl+FXzzJRiAon2vxgyFarCYBbZnPN2ikNU=; b=FCfO9AaJM2BnaoTnnBSE8lw8IQbA+rXnyR5weYsVhYSEC7F+zz9Q8tNvlR9LnVc0qcgWh+vvMsDZYzpE3Ev3wb4YmOruuCZkEgsHUJ4LQpgsefKdT439698BepR6liXAVs/h6fUAC/AlgryVi+ZH44QumVMrWs7zj+aPl8VVBJs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1680037939394696.6146610210435; Tue, 28 Mar 2023 14:12:19 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phGbL-0003S8-DU; Tue, 28 Mar 2023 17:11:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phGbH-0003R3-0h for qemu-devel@nongnu.org; Tue, 28 Mar 2023 17:11:28 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phGbE-0004sC-0i for qemu-devel@nongnu.org; Tue, 28 Mar 2023 17:11:26 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-39-cqHv2wCbMQW1O5npEvtH1g-1; Tue, 28 Mar 2023 17:11:21 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 694F92800497; Tue, 28 Mar 2023 21:11:20 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0BB6FC15BA0; Tue, 28 Mar 2023 21:11:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680037882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+gcVU8asKsl+FXzzJRiAon2vxgyFarCYBbZnPN2ikNU=; b=RJpMGf9LrULw3+HRQ8jA2+E5yaT9DWWW1YNw4MpB9junfjA65uGU+snE+fl4buMLz73ugf ZYDSAhdMawIOc/24MCd5jUmV64x5k+pa7utu2POaNeAvXC6ayboXvHyHIBuSWEoa5FII+o 8kgPIY9uCgFaNYA5agHMCd9tJ/oPUPU= X-MC-Unique: cqHv2wCbMQW1O5npEvtH1g-1 From: John Snow To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , John Snow , Wainer dos Santos Moschetta , Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Beraldo Leal , Thomas Huth , Cleber Rosa Subject: [RFC PATCH 1/3] python: add mkvenv.py Date: Tue, 28 Mar 2023 17:11:17 -0400 Message-Id: <20230328211119.2748442-2-jsnow@redhat.com> In-Reply-To: <20230328211119.2748442-1-jsnow@redhat.com> References: <20230328211119.2748442-1-jsnow@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1680037940109100015 Content-Type: text/plain; charset="utf-8" This script will be responsible for building a Python virtual environment at configure time. As such, it exists outside of the installable python packages and *must* be runnable with minimal dependencies. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 445 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 445 insertions(+) create mode 100644 python/scripts/mkvenv.py diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py new file mode 100644 index 00000000000..d48880c4205 --- /dev/null +++ b/python/scripts/mkvenv.py @@ -0,0 +1,445 @@ +""" +mkvenv - QEMU venv bootstrapping utility + +usage: TODO/FIXME +""" + +# Copyright (C) 2022-2023 Red Hat, Inc. +# +# Authors: +# John Snow +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import argparse +from importlib.util import find_spec +import logging +import os +from os import PathLike +from pathlib import Path +import re +import stat +import subprocess +import sys +import traceback +from types import SimpleNamespace +from typing import Generator, Dict, Sequence, Optional, Union, Iterator +import venv + + +# Do not add any mandatory dependencies from outside the stdlib: +# This script *must* be usable standalone! + +DirType =3D Union[str, bytes, 'PathLike[str]', 'PathLike[bytes]'] +logger =3D logging.getLogger('mkvenv') + + +class Ouch(RuntimeError): + """An Exception class we can't confuse with a builtin.""" + + +class QemuEnvBuilder(venv.EnvBuilder): + """ + An extension of venv.EnvBuilder for building QEMU's configure-time ven= v. + + The only functional change is that it adds the ability to regenerate + console_script shims for packages available via system_site + packages. + + (And a metric ton of debugging printfs) + + Parameters for base class init: + - system_site_packages: bool =3D False + - clear: bool =3D False + - symlinks: bool =3D False + - upgrade: bool =3D False + - with_pip: bool =3D False + - prompt: Optional[str] =3D None + - upgrade_deps: bool =3D False (Since 3.9) + """ + def __init__(self, *args, **kwargs) -> None: + self.script_packages =3D kwargs.pop('script_packages', ()) + super().__init__(*args, **kwargs) + # venv class is cute and toggles this off before post_setup, + # but we need it to decide if we want to generate shims or not. + self._system_site_packages =3D self.system_site_packages + # Make the context available post-creation: + self._context: Optional[SimpleNamespace] =3D None + + def ensure_directories(self, env_dir: DirType) -> SimpleNamespace: + logger.debug("ensure_directories(env_dir=3D%s)", env_dir) + context =3D super().ensure_directories(env_dir) + # Here's what's in the context blob: + logger.debug("env_dir %s", context.env_dir) + logger.debug("env_name %s", context.env_name) + logger.debug("prompt %s", context.prompt) + logger.debug("executable %s", context.executable) + logger.debug("inc_path %s", context.inc_path) + if 'lib_path' in context.__dict__: + # 3.12+ + logger.debug("lib_path %s", context.lib_path) + logger.debug("bin_path %s", context.bin_path) + logger.debug("bin_name %s", context.bin_name) + logger.debug("env_exe %s", context.env_exe) + if 'env_exec_cmd' in context.__dict__: + # 3.9+ + logger.debug("env_exec_cmd %s", context.env_exec_cmd) + self._context =3D context + return context + + def create(self, env_dir: DirType) -> None: + logger.debug("create(env_dir=3D%s)", env_dir) + super().create(env_dir) + self.post_post_setup(self._context) + + def create_configuration(self, context: SimpleNamespace) -> None: + logger.debug("create_configuration(...)") + super().create_configuration(context) + + def setup_python(self, context: SimpleNamespace) -> None: + logger.debug("setup_python(...)") + super().setup_python(context) + + def setup_scripts(self, context: SimpleNamespace) -> None: + logger.debug("setup_scripts(...)") + super().setup_scripts(context) + + # def upgrade_dependencies(self, context): ... # only Since 3.9 + + def post_setup(self, context: SimpleNamespace) -> None: + # Generate console_script entry points for system packages + # e.g. meson, sphinx, pip, etc. + logger.debug("post_setup(...)") + if self._system_site_packages: + generate_console_scripts( + context.env_exe, context.bin_path, self.script_packages) + # + # print the python executable to stdout for configure. + print(context.env_exe) + + def post_post_setup(self, context: SimpleNamespace) -> None: + # This is the very final hook that occurs *after* enabling + # system-site-packages. + subprocess.run((context.env_exe, __file__, 'checkpip'), check=3DTr= ue) + + +def need_ensurepip() -> bool: + """ + Tests for the presence of setuptools and pip. + + :return: `True` if we do not detect both packages. + """ + # Don't try to actually import them, it's fraught with danger: + # https://github.com/pypa/setuptools/issues/2993 + if find_spec("setuptools") and find_spec("pip"): + return False + return True + + +def make_venv( # pylint: disable=3Dtoo-many-arguments + venv_path: Union[str, Path], + system_site_packages: bool =3D False, + clear: bool =3D True, + symlinks: Optional[bool] =3D None, + upgrade: bool =3D False, + with_pip: Optional[bool] =3D None, + script_packages: Sequence[str]=3D(), +) -> None: + """ + Create a venv using the QemuEnvBuilder class. + + TODO: write docs O:-) + """ + logging.debug("%s: make_venv(venv_path=3D%s, system_site_packages=3D%s= , " + "clear=3D%s, upgrade=3D%s, with_pip=3D%s, script_package= s=3D%s)", + __file__, str(venv_path), system_site_packages, + clear, upgrade, with_pip, script_packages) + + print(f"MKVENV {str(venv_path)}", file=3Dsys.stderr) + + # ensurepip is slow: venv creation can be very fast for cases where + # we allow the use of system_site_packages. Toggle ensure_pip on only + # in the cases where we really need it. + if with_pip is None: + with_pip =3D True if not system_site_packages else need_ensurepip() + logger.debug("with_pip unset, choosing %s", with_pip) + + if symlinks is None: + # Default behavior of standard venv CLI + symlinks =3D os.name !=3D "nt" + + if with_pip and not find_spec("ensurepip"): + msg =3D ("Python's ensurepip module is not found.\n" + + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + + "Either install ensurepip, or alleviate the need for it in = the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + + "(Hint: Debian puts ensurepip in its python3-venv package.)= ") + raise Ouch(msg) + + builder =3D QemuEnvBuilder( + system_site_packages=3Dsystem_site_packages, + clear=3Dclear, + symlinks=3Dsymlinks, + upgrade=3Dupgrade, + with_pip=3Dwith_pip, + script_packages=3Dscript_packages, + ) + try: + logger.debug("Invoking builder.create()") + try: + builder.create(str(venv_path)) + except SystemExit as exc: + # Some versions of the venv module raise SystemExit; *nasty*! + # We want the exception that prompted it. It might be a subpro= cess + # error that has output we *really* want to see. + logger.debug("Intercepted SystemExit from EnvBuilder.create()") + raise exc.__cause__ or exc.__context__ or exc + logger.debug("builder.create() finished") + except subprocess.CalledProcessError as exc: + print(f"cmd: {exc.cmd}", file=3Dsys.stderr) + print(f"returncode: {exc.returncode}", file=3Dsys.stderr) + if exc.stdout: + print("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D stdout =3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D", file=3Dsys.stderr) + print(exc.stdout, file=3Dsys.stderr) + print("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D", file=3Dsys.stderr) + if exc.stderr: + print("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D stderr =3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D", file=3Dsys.stderr) + print(exc.stderr, file=3Dsys.stderr) + print("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D", file=3Dsys.stderr) + raise Ouch("VENV creation subprocess failed.") from exc + + +def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=3Dimport-outside-toplevel + try: + # First preference: Python 3.8+ stdlib + from importlib.metadata import ( + PackageNotFoundError, + distribution, + ) + except ImportError as exc: + logger.debug("%s", str(exc)) + # Second preference: Commonly available PyPI backport + from importlib_metadata import ( + PackageNotFoundError, + distribution, + ) + + # Borrowed from CPython (Lib/importlib/metadata/__init__.py) + pattern =3D re.compile( + r'(?P[\w.]+)\s*' + r'(:\s*(?P[\w.]+)\s*)?' + r'((?P\[.*\])\s*)?$' + ) + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + entry_points =3D distribution(package).entry_points + except PackageNotFoundError: + continue + + # The EntryPoints type is only available in 3.10+, + # treat this as a vanilla list and filter it ourselves. + entry_points =3D filter( + lambda ep: ep.group =3D=3D 'console_scripts', entry_points) + + for entry_point in entry_points: + # Python 3.8 doesn't have 'module' or 'attr' attributes + if not (hasattr(entry_point, 'module') and + hasattr(entry_point, 'attr')): + match =3D pattern.match(entry_point.value) + assert match is not None + module =3D match.group('module') + attr =3D match.group('attr') + else: + module =3D entry_point.module + attr =3D entry_point.attr + yield { + 'name': entry_point.name, + 'module': module, + 'import_name': attr, + 'func': attr, + } + + return _generator() + + +def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]= ]: + # pylint: disable=3Dimport-outside-toplevel + # Bundled with setuptools; has a good chance of being available. + import pkg_resources + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + eps =3D pkg_resources.get_entry_map(package, 'console_scri= pts') + except pkg_resources.DistributionNotFound: + continue + + for entry_point in eps.values(): + yield { + 'name': entry_point.name, + 'module': entry_point.module_name, + 'import_name': ".".join(entry_point.attrs), + 'func': ".".join(entry_point.attrs), + } + + return _generator() + + +# Borrowed/adapted from pip's vendored version of distutils: +SCRIPT_TEMPLATE =3D r'''#!{python_path:s} +# -*- coding: utf-8 -*- +import re +import sys +from {module:s} import {import_name:s} +if __name__ =3D=3D '__main__': + sys.argv[0] =3D re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) + sys.exit({func:s}()) +''' + + +def generate_console_scripts(python_path: str, bin_path: str, + packages: Sequence[str]) -> None: + """ + Generate script shims for console_script entry points in @packages. + """ + if not packages: + return + + def _get_entry_points() -> Iterator[Dict[str, str]]: + try: + return _gen_importlib(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + return _gen_pkg_resources(packages) + + try: + entry_points =3D _get_entry_points() + except ImportError as exc: + logger.debug("%s", str(exc)) + raise Ouch( + "Neither importlib.metadata nor pkg_resources found, " + "can't generate console script shims.\n" + "Use Python 3.8+, or install importlib-metadata, or setuptools= ." + ) from exc + + for entry_point in entry_points: + script_path =3D os.path.join(bin_path, entry_point['name']) + script =3D SCRIPT_TEMPLATE.format(python_path=3Dpython_path, **ent= ry_point) + with open(script_path, "w", encoding=3D'UTF-8') as file: + file.write(script) + fd =3D file.fileno() + os.chmod(fd, os.stat(fd).st_mode | stat.S_IEXEC) + logger.debug("wrote '%s'", script_path) + + +def checkpip(): + """ + Debian10 has a pip that's broken when used inside of a virtual environ= ment. + + We try to detect and correct that case here. + """ + try: + import pip._internal + logger.debug("pip appears to be working correctly.") + return + except ModuleNotFoundError as exc: + if exc.name =3D=3D 'pip._internal': + # Uh, fair enough. They did say "internal". + # Let's just assume it's fine. + return + logger.warning("pip appears to be malfunctioning: %s", str(exc)) + + # Test for ensurepip: + try: + import ensurepip + except ImportError as exc: + raise Ouch( + "pip appears to be non-functional, " + "and Python's ensurepip module is not found.\n" + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + "(Hint: Debian puts ensurepip in its python3-venv package.)" + ) from exc + + logging.debug("Attempting to repair pip ...") + subprocess.run((sys.executable, '-m', 'ensurepip'), + stdout=3Dsubprocess.DEVNULL, check=3DTrue) + logging.debug("Pip is now (hopefully) repaired!") + + +def main() -> int: + """CLI interface to make_qemu_venv. See module docstring.""" + if os.environ.get('DEBUG') or os.environ.get('GITLAB_CI'): + # You're welcome. + logging.basicConfig(level=3Dlogging.DEBUG) + + parser =3D argparse.ArgumentParser(description=3D"Bootstrap QEMU venv.= ") + subparsers =3D parser.add_subparsers( + title=3D"Commands", + description=3D"Various actions this utility can perform", + prog=3D"prog", + dest=3D"command", + required=3DTrue, + metavar=3D"command", + help=3D'Description') + + # + subparser =3D subparsers.add_parser('create', help=3D'create a venv') + subparser.add_argument( + '--gen', + type=3Dstr, + action=3D'append', + help=3D"Regenerate console_scripts for given packages, if found.", + ) + subparser.add_argument( + 'target', + type=3Dstr, + action=3D'store', + help=3D"Target directory to install virtual environment into.", + ) + + # + subparser =3D subparsers.add_parser( + 'checkpip', help=3D'test pip and fix if necessary') + + args =3D parser.parse_args() + + try: + if args.command =3D=3D 'create': + script_packages =3D [] + for element in args.gen or (): + script_packages.extend(element.split(",")) + make_venv( + args.target, + system_site_packages=3DTrue, + clear=3DTrue, + upgrade=3DFalse, + with_pip=3DNone, # Autodetermine + script_packages=3Dscript_packages, + ) + logger.debug("mkvenv.py create - exiting") + if args.command =3D=3D 'checkpip': + checkpip() + logger.debug("mkvenv.py checkpip - exiting") + except Ouch as exc: + print("\n*** Ouch! ***\n", file=3Dsys.stderr) + print(str(exc), "\n\n", file=3Dsys.stderr) + return 1 + except: # pylint: disable=3Dbare-except + print("mkvenv did not complete successfully:", file=3Dsys.stderr) + traceback.print_exc() + return 2 + return 0 + + +if __name__ =3D=3D '__main__': + sys.exit(main()) --=20 2.39.2 From nobody Tue May 14 09:22:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1680037937; cv=none; d=zohomail.com; s=zohoarc; b=kKptXevrQ2aQEdCHyXNTKkBEFEeHy5DuujZdUBjZjQpzSnd4rmvWPRA+BebtkMp/OFAY6IpbSE+ZCZHumc+JDrUNq7rwkqd4vcRCGVDrzvJoSMBPeAFUXt36F+Bha0uznDMPALfvk6RyWoLKsfNzsx42+rr4bPtgFMx+hiBlrc0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1680037937; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=BwaJRLW+gd4LyNpezKNDSLIf+CC9ZFcawv875W/9YD0=; b=JOG7K9GaQvBYBqPH/GEJUfW+iYnysiyn51pfFu0yXdr/uCmLQHKoaVLZmLdb+W09misyAvBnYdSPu4DFcByOoBK06Wyek3sGJS+hwC+/sd7xDrMKjvVYbZKL4YYqEqXKUa5SKqZIhchlusJoZqpEGeMoHCDxcqlpogVxQMzPrtY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 168003793705172.32323930302039; Tue, 28 Mar 2023 14:12:17 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phGbM-0003S9-1S; Tue, 28 Mar 2023 17:11:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phGbH-0003R5-LX for qemu-devel@nongnu.org; Tue, 28 Mar 2023 17:11:28 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phGbF-0004sg-NT for qemu-devel@nongnu.org; Tue, 28 Mar 2023 17:11:27 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-551-YgzVL0YCOsSjiH_jf-pk5w-1; Tue, 28 Mar 2023 17:11:21 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D00743810B09; Tue, 28 Mar 2023 21:11:20 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7878BC15BA0; Tue, 28 Mar 2023 21:11:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680037884; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BwaJRLW+gd4LyNpezKNDSLIf+CC9ZFcawv875W/9YD0=; b=TAON9j4frouB6aH1mLSxByipFmDnBi5W6j3XkvVqqzKzm/No3Pn3vdjTrkx06Hpej4lkNn ILJqx23eMlDRWbr25zCGTXIpOv2Wctq+94hXPhC+uWe3DCseFWH4Q406nNrDa90tSpNJhy D1VemCJMKMAM6q9o1MoHGQ40bbbjmpk= X-MC-Unique: YgzVL0YCOsSjiH_jf-pk5w-1 From: John Snow To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , John Snow , Wainer dos Santos Moschetta , Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Beraldo Leal , Thomas Huth , Cleber Rosa Subject: [RFC PATCH 2/3] tests: add python3-venv dependency Date: Tue, 28 Mar 2023 17:11:18 -0400 Message-Id: <20230328211119.2748442-3-jsnow@redhat.com> In-Reply-To: <20230328211119.2748442-1-jsnow@redhat.com> References: <20230328211119.2748442-1-jsnow@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1680037938224100007 Content-Type: text/plain; charset="utf-8" Several debian-based tests need the python3-venv dependency as a consequence of Debian debundling the "ensurepip" module normally included with Python. As this series stands right now, in most cases, Debian requires EITHER: (A) setuptools and pip, or (B) ensurepip It's quite likely most developer workstations running Debian or a Debian-based distro already have all of these packages. venv creation will actually be quite a bit faster by installing pip and setuptools, but for the purposes of this patch I am only adding the strict minimum necessary. (For that reason, I chose ensurepip to make sure that as we continue tweaking and developing the mkvenv script, we can count on Debian having a fully functional Python installation; but for the sake of this series it is an arbitrary choice here.) In one case, even though we already have pip and setuptools, we still need ensurepip because the version of these tools that shipped with Debian are known to be broken and do not function correctly inside of a venv. Using ensurepip bypasses the bug. (This is only for Debian 10, which we technically no longer support. Even so, I wanted to prove it's possible to support our oldest platforms.) Signed-off-by: John Snow --- tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++- tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/= docker/dockerfiles/debian-all-test-cross.docker index 981e9bdc7b0..f9f401544a0 100644 --- a/tests/docker/dockerfiles/debian-all-test-cross.docker +++ b/tests/docker/dockerfiles/debian-all-test-cross.docker @@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ gcc-sh4-linux-gnu \ libc6-dev-sh4-cross \ gcc-sparc64-linux-gnu \ - libc6-dev-sparc64-cross + libc6-dev-sparc64-cross \ + python3-venv =20 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hpp= a-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-u= ser,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,p= pc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc6= 4-linux-user diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/d= ocker/dockerfiles/debian-hexagon-cross.docker index 5308ccb8fe9..344eec900b0 100644 --- a/tests/docker/dockerfiles/debian-hexagon-cross.docker +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker @@ -20,7 +20,8 @@ RUN apt-get update && \ bison \ flex \ git \ - ninja-build && \ + ninja-build \ + python3-venv && \ # Install QEMU build deps for use in CI DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ apt build-dep -yy --arch-only qemu diff --git a/tests/docker/dockerfiles/debian-riscv64-cross.docker b/tests/d= ocker/dockerfiles/debian-riscv64-cross.docker index 803afb95734..081404e0141 100644 --- a/tests/docker/dockerfiles/debian-riscv64-cross.docker +++ b/tests/docker/dockerfiles/debian-riscv64-cross.docker @@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata apt instal= l -yy \ libglib2.0-dev \ ninja-build \ pkg-config \ - python3 + python3 \ + python3-venv =20 # Add ports and riscv64 architecture RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/= apt/sources.list diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/d= ocker/dockerfiles/debian-tricore-cross.docker index cfd2faf9a80..269bfa8d423 100644 --- a/tests/docker/dockerfiles/debian-tricore-cross.docker +++ b/tests/docker/dockerfiles/debian-tricore-cross.docker @@ -33,7 +33,8 @@ RUN apt update && \ pkgconf \ python3-pip \ python3-setuptools \ - python3-wheel + python3-wheel \ + python3-venv =20 RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download= /tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \ | tar -xzC /usr/local/ --=20 2.39.2 From nobody Tue May 14 09:22:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1680037937; cv=none; d=zohomail.com; s=zohoarc; b=ezfux4pz/HgTx75Atk9KNXKkx161RZ5UoO0Re7Qdd0kybp+oVd24F43HI0Wpq0uxospqbSmOJubqxd3EKaUoWuJTzPZ5K0qAPxc6+04y1kjcdYtNMnxIDsnHkyRBvQrvzElotkpkAKh8mI4iGlDg8xkBsnYqyTTPkR2jwMkH/5g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1680037937; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=bvbjVftZREXEcr3Z3LmWGSA++itr8KSo6XaHIAINdts=; b=KoOQrEleAqN9U7rxnmiCS/FMiBydIIkcYTbvp4z8FiEx2oB0FfkhXRB+F4QBsSOjToFgCprBPgx5CdlquLm+y+wdSNrvSFP+Z33KFr2ZsANuwhoQEojTlrBu2hm7eIfC99MvQNnICsQohGf+xax3z9G6oxoEJFVSVaL+JaRxVF0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1680037937050507.54479103255073; Tue, 28 Mar 2023 14:12:17 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phGbJ-0003Rg-Ok; Tue, 28 Mar 2023 17:11:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phGbI-0003R8-2g for qemu-devel@nongnu.org; Tue, 28 Mar 2023 17:11:28 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phGbF-0004sl-Ns for qemu-devel@nongnu.org; Tue, 28 Mar 2023 17:11:27 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-553-gThBAK1zPD2biSH9c-QGOw-1; Tue, 28 Mar 2023 17:11:21 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 414651C068D9; Tue, 28 Mar 2023 21:11:21 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD597C15BA0; Tue, 28 Mar 2023 21:11:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680037885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bvbjVftZREXEcr3Z3LmWGSA++itr8KSo6XaHIAINdts=; b=gr5bFpSVYBrxitqIep4LiOfbOziLQ4HeGqVZZX/wMVty/nI63a3EubxrL+TCnsLKsh+0ej ryzAxtxLCSohMRzLhH+CWEeHYbP50UU/OYEHjv1Kv5ls0Y2VjLFBRZrtkYx06wimp8Go+X ptf4BvlWWihNMyEaGbMEMHyvMClap+c= X-MC-Unique: gThBAK1zPD2biSH9c-QGOw-1 From: John Snow To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , John Snow , Wainer dos Santos Moschetta , Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Beraldo Leal , Thomas Huth , Cleber Rosa Subject: [RFC PATCH 3/3] configure: install meson to a python virtual environment Date: Tue, 28 Mar 2023 17:11:19 -0400 Message-Id: <20230328211119.2748442-4-jsnow@redhat.com> In-Reply-To: <20230328211119.2748442-1-jsnow@redhat.com> References: <20230328211119.2748442-1-jsnow@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1680037938287100009 Content-Type: text/plain; charset="utf-8" This patch changes how we detect and install meson. This patch creates a lightweight Python virtual environment using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson. In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from source into the newly created Python virtual environment. At present, the source is obtained in the same manner as it has been; preferring git submodules first and relying on vendored source as a backup. This patch does *not* connect to PyPI and will work offline for all supported build platforms. Future improvements to this patch may add optional PyPI support as a replacement for git submodules. As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously. Signed-off-by: John Snow --- configure | 155 +++++++++++++++++++--------- .gitlab-ci.d/buildtest-template.yml | 2 +- 2 files changed, 108 insertions(+), 49 deletions(-) diff --git a/configure b/configure index 05bed4f4a11..1ec054775f6 100755 --- a/configure +++ b/configure @@ -607,7 +607,6 @@ check_py_version() { python=3D first_python=3D if test -z "${PYTHON}"; then - explicit_python=3Dno # A bare 'python' is traditionally python 2.x, but some distros # have it as python 3.x, so check in both places. for binary in python3 python python3.11 python3.10 python3.9 python3.8= python3.7 python3.6; do @@ -626,7 +625,6 @@ else # Same as above, but only check the environment variable. has "${PYTHON}" || error_exit "The PYTHON environment variable does no= t point to an executable" python=3D$(command -v "$PYTHON") - explicit_python=3Dyes if check_py_version "$python"; then # This one is good. first_python=3D @@ -718,7 +716,7 @@ for opt do ;; --install=3D*) ;; - --python=3D*) python=3D"$optarg" ; explicit_python=3Dyes + --python=3D*) python=3D"$optarg" ;; --skip-meson) skip_meson=3Dyes ;; @@ -1078,61 +1076,122 @@ if ! check_py_version "$python"; then "Use --python=3D/path/to/python to specify a supported Python." fi =20 -# Resolve PATH + suppress writing compiled files -python=3D"$(command -v "$python") -B" - -has_meson() { - local python_dir=3D$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; = then - # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi +# Resolve PATH +python=3D"$(command -v "$python")" +# This variable is intended to be used only for error messages: +target_python=3D$python + +# Create a Python virtual environment using our configured python. +# The stdout of this script will be the location of a symlink that +# points to the configured Python. +# Entry point scripts for pip, meson, and sphinx are generated if those +# packages are present. + +# Defaults assumed for now: +# - venv is cleared if it exists already; +# - venv is allowed to use system packages; +# - all setup is performed **offline**; +# - No packages are installed by default; +# - pip is not installed into the venv when possible, +# but ensurepip is called as a fallback when necessary. + +echo "python determined to be '$python'" +echo "python version: $($python --version)" + +echo " =3D=3D=3D pip list (host) =3D=3D=3D " +$python -m pip list --disable-pip-version-check +echo " =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D " +echo "" + +python=3D"$($python -B "${source_path}/python/scripts/mkvenv.py" create --= gen pip,meson,sphinx pyvenv)" +if test "$?" -ne 0 ; then + error_exit "python venv creation failed" +fi + +# Suppress writing compiled files +python=3D"$python -B" +echo "Configured python as '$python'" + +echo " =3D=3D=3D pip list (venv) =3D=3D=3D " +$python -m pip list --disable-pip-version-check +echo " =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D " +echo "" + +pip_install() { + $python -m pip install -v \ + --disable-pip-version-check \ + --no-index \ + "$@" } =20 -if test -z "$meson"; then - if test "$explicit_python" =3D no && has_meson && version_ge "$(meson = --version)" 0.61.5; then - meson=3Dmeson - elif test "$git_submodules_action" !=3D 'ignore' ; then - meson=3Dgit - elif test -e "${source_path}/meson/meson.py" ; then - meson=3Dinternal +# OK, let's have some fun! + +# This install command is intended to either fail or be a NOP; +# because we're offline, it's just a convenient version check. +if ! pip_install 'meson>=3D0.61.5'; then + # Either we don't have Meson, or our Meson is too old. + # (Future revisions of this patchset can be less chatty.) + if test -e pyvenv/bin/meson; then + echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)" + elif has meson ; then + echo "Meson was found installed on your system," \ + "but not for the configured Python interpreter ($target_pytho= n)." + echo "(Hint: check '$(which meson)' to see which interpreter its s= hebang uses.)" else - if test "$explicit_python" =3D yes; then - error_exit "--python requires using QEMU's embedded Meson dist= ribution, but it was not found." - else - error_exit "Meson not found. Use --meson=3D/path/to/meson" + echo "Meson was not found." + fi + + # OK, but can we fix it, though? :~) + if test "$git_submodules_action" !=3D 'ignore' ; then + git_submodules=3D"${git_submodules} meson" + echo "Attempting to install meson from git submodule ..." + # Stolen from later in the configure file. + # Is it a problem if we front-load this now and run it again later? + if ! (GIT=3D"$git" "$source_path/scripts/git-submodule.sh" "$git_s= ubmodules_action" "$git_submodules"); then + exit 1 fi + elif test -e "${source_path}/meson/setup.cfg" ; then + echo "Attempting to install meson from vendored source ..." + else + # In the future, we could use PyPI as a source if the user allows = it. + # For now, you're outta luck! + error_exit "A suitable version of Meson was not found." fi -else - # Meson uses its own Python interpreter to invoke other Python scripts, - # but the user wants to use the one they specified with --python. + + # If we're here, we have the meson source and we can attempt to + # install it into our venv. + + # We want to install meson with --no-use-pep517 if possible, + # because it avoids needing a 'wheel' dependency. Old versions + # of pip do this by default, so test for the behavior. # - # We do not want to override the distro Python interpreter (and someti= mes - # cannot: for example in Homebrew /usr/bin/meson is a bash script), so - # just require --meson=3Dgit|internal together with --python. - if test "$explicit_python" =3D yes; then - case "$meson" in - git | internal) ;; - *) error_exit "--python requires using QEMU's embedded Meson d= istribution." ;; - esac + # --no-build-isolation was added to pip 10.0. + # --no-use-pep517 was added ... sometime after 18.1? + pip_flags=3D'--no-build-isolation' + if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>= &1 ; then + pip_flags=3D"${pip_flags} --no-use-pep517" + fi + if ! pip_install $pip_flags "${source_path}/meson" ; then + exit 1 fi fi =20 -if test "$meson" =3D git; then - git_submodules=3D"${git_submodules} meson" -fi +# At this point, we expect Meson to be installed and available. +# We expect mkvenv or pip to have created pyvenv/bin/meson for us. +# We ignore PATH completely here: we want to use the venv's Meson +# *exclusively*. +meson=3D"$(cd pyvenv/bin; pwd)/meson" =20 -case "$meson" in - git | internal) - meson=3D"$python ${source_path}/meson/meson.py" - ;; - *) meson=3D$(command -v "$meson") ;; -esac +# Smoke tests: these shouldn't fail, but I have little faith in esoteric +# platforms I've never used to not do something Deep Strange when I am +# not looking. +if ! test -e "$meson" ; then + error_exit "??? pyvenv/bin/meson not found, somehow ???" +fi +echo "Meson $($meson --version) found: $meson" +if ! version_ge "$($meson --version)" 0.61.5; then + error_exit "??? Meson version is too old ???" +fi =20 # Probe for ninja =20 diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-t= emplate.yml index a6cfe9be970..6cfe644d8fe 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -17,7 +17,7 @@ { cat config.log meson-logs/meson-log.txt && exit 1; } - if test -n "$LD_JOBS"; then - ../meson/meson.py configure . -Dbackend_max_links=3D"$LD_JOBS" ; + pyvenv/bin/meson configure . -Dbackend_max_links=3D"$LD_JOBS" ; fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS"; --=20 2.39.2