This patch series creates a mandatory python virtual environment
("venv") during configure time and uses it to ensure the availability of
meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for motivations. The summary
is that the goal of this series is to ensure that the `python` used to
run meson is the same `python` used to run Sphinx, tests, and any
build-time python scripting we have. As it stands, meson and sphinx (and
their extensions) *may* run in a different python environment than the
one configured and chosen by the user at configure/build time.
The effective change of this series is that QEMU will now
unconditionally create a venv at configure-time and will ensure that
meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified
by configure. (Which in turn is always the one specified by --python
or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build
system, meson, sphinx, avocado tests, vm tests or CI are always
executed within this venv.
(iotests are not yet integrated; I plan to tackle this separately as a
follow-up in order to have a more tightly focused scope on that
series.)
- It remains possible to build and test fully offline.
(In most cases, you just need meson and sphinx from your distro's repo.)
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever
possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI
versions for speed, repeatability and ensuring tarball builds work
as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed
on-demand automatically to make developer environments "just work".
- Works for Python 3.6 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an
asterisk.)
- The meson git submodule is unused after this series and can be removed.
... I could write more, and I'm sure there's a few things that need
buffing out or changing here and there, but I think it's largely good.
So, enjoy!
~js
John Snow (22):
python: update pylint configuration
python: add mkvenv.py
mkvenv: add better error message for missing pyexpat module
mkvenv: add nested venv workaround
mkvenv: add ensure subcommand
mkvenv: add diagnose() method for ensure() failures
mkvenv: add console script entry point generation
mkvenv: work around broken pip installations on Debian 10
tests/docker: add python3-venv dependency
tests/vm: Configure netbsd to use Python 3.10
tests/vm: add py310-expat to NetBSD
python: add vendor.py utility
configure: create a python venv unconditionally
python/wheels: add vendored meson package
configure: use 'mkvenv ensure meson' to bootstrap meson
qemu.git: drop meson git submodule
tests: Use configure-provided pyvenv for tests
configure: move --enable-docs and --disable-docs back to configure
configure: bootstrap sphinx with mkvenv
configure: add --enable-pypi and --disable-pypi
configure: Add courtesy hint to Python version failure message
mkvenv.py: experiment; use distlib to generate script entry points
Paolo Bonzini (5):
python: shut up "pip install" during "make check-minreqs"
mkvenv: create pip binary in virtual environment
Python: Drop support for Python 3.6
mkvenv: mark command as required
python: bump some of the dependencies
docs/about/build-platforms.rst | 2 +-
docs/conf.py | 9 -
docs/devel/acpi-bits.rst | 6 +-
docs/devel/testing.rst | 14 +-
docs/meson.build | 2 +-
configure | 155 ++--
.gitlab-ci.d/buildtest-template.yml | 4 +-
.gitlab-ci.d/buildtest.yml | 6 +-
.gitmodules | 3 -
meson | 1 -
meson_options.txt | 2 -
python/Makefile | 19 +-
python/scripts/mkvenv.py | 858 ++++++++++++++++++
python/scripts/vendor.py | 74 ++
python/setup.cfg | 27 +-
python/tests/flake8.sh | 1 +
python/tests/isort.sh | 1 +
python/tests/minreqs.txt | 16 +-
python/tests/mypy.sh | 1 +
python/tests/pylint.sh | 1 +
python/wheels/meson-0.61.5-py3-none-any.whl | Bin 0 -> 862509 bytes
.../org.centos/stream/8/x86_64/test-avocado | 4 +-
scripts/device-crash-test | 2 +-
scripts/meson-buildoptions.sh | 3 -
scripts/qapi/mypy.ini | 2 +-
tests/Makefile.include | 10 +-
.../dockerfiles/debian-all-test-cross.docker | 3 +-
.../dockerfiles/debian-hexagon-cross.docker | 3 +-
.../dockerfiles/debian-riscv64-cross.docker | 3 +-
.../dockerfiles/debian-tricore-cross.docker | 3 +-
tests/requirements.txt | 7 +-
tests/vm/netbsd | 2 +
32 files changed, 1108 insertions(+), 136 deletions(-)
delete mode 160000 meson
create mode 100644 python/scripts/mkvenv.py
create mode 100755 python/scripts/vendor.py
create mode 100644 python/wheels/meson-0.61.5-py3-none-any.whl
--
2.40.0