On Tue, May 16, 2023 at 6:57 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> This patch series creates a mandatory python virtual environment
> ("venv") during configure time and uses it to ensure the availability of
> meson and sphinx.
>
> See https://www.qemu.org/2023/03/24/python/ for motivations. The summary
> is that the goal of this series is to ensure that the `python` used to
> run meson is the same `python` used to run Sphinx, tests, and any
> build-time python scripting we have. As it stands, meson and sphinx (and
> their extensions) *may* run in a different python environment than the
> one configured and chosen by the user at configure/build time.
>
> The effective change of this series is that QEMU will now
> unconditionally create a venv at configure-time and will ensure that
> meson (and sphinx, if docs are enabled) are available through that venv.
>
> Some important points as a pre-emptive "FAQ":
>
> - Works for Python 3.6 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
> Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> No new dependencies (...for most platforms. Debian and NetBSD get an
> asterisk, see patch 4).
>
> - The venv is unconditionally created and lives at {build_dir}/pyvenv.
> The python interpreter used by this venv is always the one identified
> by configure. (Which in turn is always the one specified by --python
> or $PYTHON). --meson and --sphinx-build disappear.
>
> - *almost* all python scripts in qemu.git executed as part of the build
> system, meson, sphinx, avocado tests, vm tests or CI are always
> executed within this venv. iotests are not yet integrated.
>
> - Missing dependencies, when possible, are fetched and installed
> on-demand automatically to make developer environments "just work".
> However, it remains possible to build and test fully offline. Right
> now, online behavior is only triggered by --enable-docs, and even
> in that case...
>
> - ... distribution packaged 'meson' and 'sphinx' are still utilized
> whenever possible as the highest preference.
>
> - The bundled 'meson' takes the shape of a .whl file, replacing the
> git submodule. qemu-qmp will follow suit, while Sphinx will not be
> bundled just like it already isn't.
>
> - about 13% of the mkvenv.py script is devoted to supporting Python
> 3.7 and Debian 10. ¯\_(ツ)_/¯
>
I like that you measured this. It feels like more than 13%, but I'm
going to trust this oddly specific number.
> Tested with "make check-minreqs" after every patch and with GitLab at
> https://gitlab.com/bonzini/qemu/-/pipelines/868627762.
For v1, I also tested with "make check-minreqs && make check-tox" and
also started checking/formatting with black --line-length=79. I
haven't added black into the linters yet because there's surely a ton
of code that needs observed before I do that, but Dan suggested it and
I liked it enough to start using it myself to take the ambiguity out
of how to handle long lines.
(Not worth a respin if black isn't happy, I didn't check. Just mentioning it.)
>
> Enjoy!
>
> John and Paolo
>
> v1->v2:
> - integrate distlib for script generation and also to look up
> installed packages
Thanks :)
> - print a more precise error for --diagnose when running in an
> isolated venv
> - reorganized code to avoid ensurepip, so that it is not spread
> across multiple patches
> - made non-fatal errors a bit less scary
"Oops" => "Oopsies!!"
>
>
> John Snow (23):
> python: update pylint configuration
> python: add mkvenv.py
> mkvenv: add better error message for broken or missing ensurepip
> mkvenv: add nested venv workaround
> mkvenv: add ensure subcommand
> mkvenv: add --diagnose option to explain "ensure" failures
> mkvenv: add console script entry point generation
> mkvenv: use pip's vendored distlib as a fallback
> mkvenv: avoid ensurepip if pip is installed
> mkvenv: work around broken pip installations on Debian 10
> tests/docker: add python3-venv dependency
> tests/vm: Configure netbsd to use Python 3.10
> tests/vm: add py310-expat to NetBSD
> python: add vendor.py utility
> configure: create a python venv unconditionally
> python/wheels: add vendored meson package
> configure: use 'mkvenv ensure meson' to bootstrap meson
> qemu.git: drop meson git submodule
> tests: Use configure-provided pyvenv for tests
> configure: move --enable-docs and --disable-docs back to configure
> configure: bootstrap sphinx with mkvenv
> configure: add --enable-pypi and --disable-pypi
> configure: Add courtesy hint to Python version failure message
>
> Paolo Bonzini (4):
> python: shut up "pip install" during "make check-minreqs"
> Python: Drop support for Python 3.6
> mkvenv: mark command as required
> python: bump some of the dependencies
>
> .gitlab-ci.d/buildtest-template.yml | 4 +-
> .gitlab-ci.d/buildtest.yml | 6 +-
> .gitmodules | 3 -
> configure | 152 +--
> docs/about/build-platforms.rst | 2 +-
> docs/conf.py | 9 -
> docs/devel/acpi-bits.rst | 6 +-
> docs/devel/testing.rst | 14 +-
> docs/meson.build | 2 +-
> meson | 1 -
> meson_options.txt | 2 -
> python/Makefile | 19 +-
> python/scripts/mkvenv.py | 897 ++++++++++++++++++
> python/scripts/vendor.py | 74 ++
> python/setup.cfg | 46 +-
> python/tests/flake8.sh | 1 +
> python/tests/isort.sh | 1 +
> python/tests/minreqs.txt | 19 +-
> python/tests/mypy.sh | 1 +
> python/tests/pylint.sh | 1 +
> python/wheels/meson-0.61.5-py3-none-any.whl | Bin 0 -> 862509 bytes
> .../org.centos/stream/8/x86_64/test-avocado | 4 +-
> scripts/device-crash-test | 2 +-
> scripts/meson-buildoptions.sh | 3 -
> scripts/qapi/mypy.ini | 2 +-
> tests/Makefile.include | 10 +-
> .../dockerfiles/debian-all-test-cross.docker | 3 +-
> .../dockerfiles/debian-hexagon-cross.docker | 3 +-
> .../dockerfiles/debian-riscv64-cross.docker | 3 +-
> .../dockerfiles/debian-tricore-cross.docker | 3 +-
> tests/requirements.txt | 7 +-
> tests/vm/netbsd | 2 +
> 32 files changed, 1167 insertions(+), 135 deletions(-)
> delete mode 160000 meson
> create mode 100644 python/scripts/mkvenv.py
> create mode 100755 python/scripts/vendor.py
> create mode 100644 python/wheels/meson-0.61.5-py3-none-any.whl
>
> --
> 2.40.1
>