From nobody Tue Dec 16 07:06:27 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1681451785; cv=none; d=zohomail.com; s=zohoarc; b=Lv3EjuSoITacbDj86pgYrca+riFAXyHDQokEcUoUfY7PLnO+gQONLoTxjY7Sv8UADOXOMVYSTXIgsrS82MVdX3wgwMoiDDYlYecD8t3zMTZXkYRCNztmEOxcEBF/IXkwo1BnKsca4nSIaJTNBoo61y3m443qe9HMqYsBzkXJgx4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1681451785; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Hoi18Z99d/5Y09VoN24QIWuhDsmKwXiy9nDSUT3ER8U=; b=TNwg2SEmIRBdlPBrCOIrcFObC4WC4WDRn4AzgtAjW/dzVVzHKMvuL/pMJk6YJFK9r+0h0stEnQPrn34ecdsm6qkNDLAoqbKTWc7N4QBqUR6BAm7R4asFGWYLDST2dfz4+c3R2xhOYrY/h4/gsjV3rWx6FQiFIfwIAetlu4/waG8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1681451785392323.1986152972448; Thu, 13 Apr 2023 22:56:25 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCP6-0007Qf-OP; Fri, 14 Apr 2023 01:55:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOz-0007Od-Li for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOk-0000Ju-94 for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:17 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-434-iLZK1QPdNiWhTof4_J0xag-1; Fri, 14 Apr 2023 01:54:56 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D2C58101A54F; Fri, 14 Apr 2023 05:54:55 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4DA75C1602A; Fri, 14 Apr 2023 05:54:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Hoi18Z99d/5Y09VoN24QIWuhDsmKwXiy9nDSUT3ER8U=; b=h3y9dD+WxsWdVnkZ8vkgdCZQ98MFqn2L/uEBFf6zHB5krYj5YNbknoyAF4xXlxCIxDH0Cg aG+zo5KaQmm634khJO3b74GFVpu9UXmfuPK0zej/ceZSfek2T/xQFYRBh0NMixPfqwSt3L /2UFUW2hX6TH0YOXSndqu7Z0tKTL/NE= X-MC-Unique: iLZK1QPdNiWhTof4_J0xag-1 From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Subject: [RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv Date: Fri, 14 Apr 2023 01:54:48 -0400 Message-Id: <20230414055449.4028284-10-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1681451787025100001 Content-Type: text/plain; charset="utf-8" This patch changes how we detect and install meson. The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson. In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from source into the newly created Python virtual environment. At present, the source is obtained in the same manner as it has been; preferring git submodules first and relying on vendored source as a backup. This patch (as of now) does *not* connect to PyPI and will work offline for all supported build platforms. As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously. Notes: - The meson git submodule can be removed in favor of just loading meson from PyPI; but we probably want a configure flag to toggle online/offline behavior. (What do we want the default to be? Online is my guess.) - Installing meson from the source tree for vendored cases (tarball releases) can be replaced by vendoring a .whl file instead. This will remove some of the `--no-use-pep517` hackiness and alleviates the requirement that users install the python3 'wheel' dependency. - Most of this logic can be moved into mkvenv.py, *especially* if the meson git submodule is removed. Signed-off-by: John Snow --- configure | 104 +++++++++++++++------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 59 insertions(+), 49 deletions(-) diff --git a/configure b/configure index 03278fd891..c8f73d4ff7 100755 --- a/configure +++ b/configure @@ -721,8 +721,6 @@ for opt do ;; --skip-meson) skip_meson=3Dyes ;; - --meson=3D*) meson=3D"$optarg" - ;; --ninja=3D*) ninja=3D"$optarg" ;; --smbd=3D*) smbd=3D"$optarg" @@ -1006,7 +1004,6 @@ Advanced options (experts only): --cross-prefix-ARCH=3DPREFIX cross compiler prefix when building ARCH gu= est test cases --make=3DMAKE use specified make [$make] --python=3DPYTHON use specified python [$python] - --meson=3DMESON use specified meson [$meson] --ninja=3DNINJA use specified ninja [$ninja] --smbd=3DSMBD use specified smbd [$smbd] --with-git=3DGIT use specified git [$git] @@ -1079,7 +1076,8 @@ fi =20 # Resolve PATH python=3D"$(command -v "$python")" -explicit_python=3Dyes +# This variable is intended to be used only for error messages: +target_python=3D$python =20 # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that @@ -1106,58 +1104,70 @@ fi # Suppress writing compiled files python=3D"$python -B" =20 -has_meson() { - local python_dir=3D$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; = then - # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi +pip_install() { + $python -m pip install -v \ + --disable-pip-version-check \ + --no-index \ + "$@" } =20 -if test -z "$meson"; then - if test "$explicit_python" =3D no && has_meson && version_ge "$(meson = --version)" 0.61.5; then - meson=3Dmeson - elif test "$git_submodules_action" !=3D 'ignore' ; then - meson=3Dgit - elif test -e "${source_path}/meson/meson.py" ; then - meson=3Dinternal +# OK, let's have some fun! + +# This install command is intended to either fail or be a NOP; +# because we're offline, it's just a convenient version check. +if ! pip_install 'meson>=3D0.61.5'; then + # Either we don't have Meson, or our Meson is too old. + # (Future revisions of this patchset can be less chatty.) + if test -e pyvenv/bin/meson; then + echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)" + elif has meson ; then + echo "Meson was found installed on your system," \ + "but not for the configured Python interpreter ($target_pytho= n)." + echo "(Hint: check '$(which meson)' to see which interpreter its s= hebang uses.)" else - if test "$explicit_python" =3D yes; then - error_exit "--python requires using QEMU's embedded Meson dist= ribution, but it was not found." - else - error_exit "Meson not found. Use --meson=3D/path/to/meson" + echo "Meson was not found." + fi + + # OK, but can we fix it, though? :~) + if test "$git_submodules_action" !=3D 'ignore' ; then + git_submodules=3D"${git_submodules} meson" + echo "Attempting to install meson from git submodule ..." + # Stolen from later in the configure file. + # Is it a problem if we front-load this now and run it again later? + if ! (GIT=3D"$git" "$source_path/scripts/git-submodule.sh" "$git_s= ubmodules_action" "$git_submodules"); then + exit 1 fi + elif test -e "${source_path}/meson/setup.cfg" ; then + echo "Attempting to install meson from vendored source ..." + else + # In the future, we could use PyPI as a source if the user allows = it. + # For now, you're outta luck! + error_exit "A suitable version of Meson was not found." fi -else - # Meson uses its own Python interpreter to invoke other Python scripts, - # but the user wants to use the one they specified with --python. + + # If we're here, we have the meson source and we can attempt to + # install it into our venv. + + # We want to install meson with --no-use-pep517 if possible, + # because it avoids needing a 'wheel' dependency. Old versions + # of pip do this by default, so test for the behavior. # - # We do not want to override the distro Python interpreter (and someti= mes - # cannot: for example in Homebrew /usr/bin/meson is a bash script), so - # just require --meson=3Dgit|internal together with --python. - if test "$explicit_python" =3D yes; then - case "$meson" in - git | internal) ;; - *) error_exit "--python requires using QEMU's embedded Meson d= istribution." ;; - esac + # --no-build-isolation was added to pip 10.0. + # --no-use-pep517 was added ... sometime after 18.1? + pip_flags=3D'--no-build-isolation' + if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>= &1 ; then + pip_flags=3D"${pip_flags} --no-use-pep517" + fi + if ! pip_install $pip_flags "${source_path}/meson" ; then + exit 1 fi fi =20 -if test "$meson" =3D git; then - git_submodules=3D"${git_submodules} meson" -fi - -case "$meson" in - git | internal) - meson=3D"$python ${source_path}/meson/meson.py" - ;; - *) meson=3D$(command -v "$meson") ;; -esac +# At this point, we expect Meson to be installed and available. +# We expect mkvenv or pip to have created pyvenv/bin/meson for us. +# We ignore PATH completely here: we want to use the venv's Meson +# *exclusively*. +meson=3D"$(cd pyvenv/bin; pwd)/meson" =20 # Probe for ninja =20 diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-t= emplate.yml index a6cfe9be97..7edb50b760 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -12,12 +12,12 @@ - mkdir build - cd build - ../configure --enable-werror --disable-docs --enable-fdt=3Dsystem - ${LD_JOBS:+--meson=3Dgit} ${TARGETS:+--target-list=3D"$TARGETS"} + ${TARGETS:+--target-list=3D"$TARGETS"} $CONFIGURE_ARGS || { cat config.log meson-logs/meson-log.txt && exit 1; } - if test -n "$LD_JOBS"; then - ../meson/meson.py configure . -Dbackend_max_links=3D"$LD_JOBS" ; + pyvenv/bin/meson configure . -Dbackend_max_links=3D"$LD_JOBS" ; fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS"; --=20 2.39.2