[PATCH v2 01/12] python: convert packages to PEP517/pyproject.toml

John Snow posted 12 patches 5 months ago
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, John Snow <jsnow@redhat.com>, Peter Maydell <peter.maydell@linaro.org>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, Markus Armbruster <armbru@redhat.com>, Michael Roth <michael.roth@amd.com>, Cleber Rosa <crosa@redhat.com>, Gerd Hoffmann <kraxel@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, Eduardo Habkost <eduardo@habkost.net>, Maksim Davydov <davydov-max@yandex-team.ru>, Richard Henderson <richard.henderson@linaro.org>, Ani Sinha <anisinha@redhat.com>, Marcelo Tosatti <mtosatti@redhat.com>, Alexander Bulekov <alxndr@bu.edu>, Bandan Das <bsd@redhat.com>, Stefan Hajnoczi <stefanha@redhat.com>, Fabiano Rosas <farosas@suse.de>, Darren Kenny <darren.kenny@oracle.com>, Qiuhao Li <Qiuhao.Li@outlook.com>, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>, Alexandre Iooss <erdnaxe@crans.org>, Mahmoud Mandour <ma.mandourr@gmail.com>, Pierrick Bouvier <pierrick.bouvier@linaro.org>, Mads Ynddal <mads@ynddal.dk>, Fam Zheng <fam@euphon.net>, Manos Pitsidianakis <manos.pitsidianakis@linaro.org>, Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>, Peter Xu <peterx@redhat.com>, Brian Cain <brian.cain@oss.qualcomm.com>, Alessandro Di Federico <ale@rev.ng>, Anton Johansson <anjo@rev.ng>, "Cédric Le Goater" <clg@kaod.org>, Steven Lee <steven_lee@aspeedtech.com>, Troy Lee <leetroy@gmail.com>, Jamin Lin <jamin_lin@aspeedtech.com>, Andrew Jeffery <andrew@codeconstruct.com.au>, Joel Stanley <joel@jms.id.au>, Bernhard Beschow <shentey@gmail.com>, Radoslaw Biernacki <rad@semihalf.com>, Leif Lindholm <leif.lindholm@oss.qualcomm.com>, Eric Auger <eric.auger@redhat.com>, Alistair Francis <alistair@alistair23.me>, "Edgar E. Iglesias" <edgar.iglesias@gmail.com>, "Michael S. Tsirkin" <mst@redhat.com>, Laurent Vivier <laurent@vivier.eu>, Antony Pavlov <antonynpavlov@gmail.com>, Beniamino Galvani <b.galvani@gmail.com>, Strahinja Jankovic <strahinja.p.jankovic@gmail.com>, Subbaraya Sundeep <sundeep.lkml@gmail.com>, Niek Linnenbank <nieklinnenbank@gmail.com>, Tyrone Ting <kfting@nuvoton.com>, Hao Wu <wuhaotsh@google.com>, Igor Mitsyanko <i.mitsyanko@gmail.com>, Michael Rolnik <mrolnik@gmail.com>, Marcel Apfelbaum <marcel.apfelbaum@gmail.com>, Yanan Wang <wangyanan55@huawei.com>, Zhao Liu <zhao1.liu@intel.com>, Helge Deller <deller@gmx.de>, Jason Wang <jasowang@redhat.com>, Yi Liu <yi.l.liu@intel.com>, "Clément Mathieu--Drif" <clement.mathieu--drif@eviden.com>, Song Gao <gaosong@loongson.cn>, Aurelien Jarno <aurelien@aurel32.net>, Huacai Chen <chenhuacai@kernel.org>, Jiaxun Yang <jiaxun.yang@flygoat.com>, Elena Ufimtseva <elena.ufimtseva@oracle.com>, Jagannathan Raman <jag.raman@oracle.com>, Akihiko Odaki <odaki@rsg.ci.i.u-tokyo.ac.jp>, Sriram Yagnaraman <sriram.yagnaraman@ericsson.com>, Jia Liu <proljc@gmail.com>, Nicholas Piggin <npiggin@gmail.com>, Daniel Henrique Barboza <danielhb413@gmail.com>, Harsh Prateek Bora <harshpb@linux.ibm.com>, Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>, "Frédéric Barrat" <fbarrat@linux.ibm.com>, "Hervé Poussineau" <hpoussin@reactos.org>, BALATON Zoltan <balaton@eik.bme.hu>, Palmer Dabbelt <palmer@dabbelt.com>, Weiwei Li <liwei1518@gmail.com>, Liu Zhiwei <zhiwei_liu@linux.alibaba.com>, Yoshinori Sato <ysato@users.sourceforge.jp>, Halil Pasic <pasic@linux.ibm.com>, Christian Borntraeger <borntraeger@linux.ibm.com>, Eric Farman <farman@linux.ibm.com>, Nina Schoetterl-Glausch <nsg@linux.ibm.com>, Magnus Damm <magnus.damm@gmail.com>, David Hildenbrand <david@redhat.com>, David Woodhouse <dwmw2@infradead.org>, Paul Durrant <paul@xen.org>, Max Filippov <jcmvbkbc@gmail.com>, Kevin Wolf <kwolf@redhat.com>, Hanna Reitz <hreitz@redhat.com>, Eric Blake <eblake@redhat.com>, Hyman Huang <yong.huang@smartx.com>, Ilya Leoshkevich <iii@linux.ibm.com>, Warner Losh <imp@bsdimp.com>, Kyle Evans <kevans@freebsd.org>, Ed Maste <emaste@freebsd.org>, Li-Wen Hsu <lwhsu@freebsd.org>
[PATCH v2 01/12] python: convert packages to PEP517/pyproject.toml
Posted by John Snow 5 months ago
Newer versions of setuptools increasingly expect that packages are
defined using the pyproject.toml/PEP517 packaging layout format. With
3.9 as our minimum, I believe it's finally appropriate to make the shift
away from the legacy packaging format.

Update documentation and dependencies that change as a result of the
different build/packaging/installation pathways.

This change has the effect of fixing "make check-dev", which has been
broken on newer versions of Fedora for a while, now.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst        | 33 ++++++++++++++++-----------------
 python/Makefile          | 18 +++++++++---------
 python/pyproject.toml    | 10 ++++++++++
 python/setup.py          | 40 ----------------------------------------
 python/tests/minreqs.txt |  2 +-
 5 files changed, 36 insertions(+), 67 deletions(-)
 create mode 100644 python/pyproject.toml
 delete mode 100755 python/setup.py

diff --git a/python/README.rst b/python/README.rst
index d62e71528d2..befa84e3261 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -5,24 +5,23 @@ This directory houses Python tooling used by the QEMU project to build,
 configure, and test QEMU. It is organized by namespace (``qemu``), and
 then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
 
-``setup.py`` is used by ``pip`` to install this tooling to the current
-environment. ``setup.cfg`` provides the packaging configuration used by
-``setup.py``. You will generally invoke it by doing one of the following:
+``pyproject.toml`` and ``setup.cfg`` are used by ``pip`` to install this
+tooling to the current environment. ``setup.cfg`` provides the packaging
+configuration, while ``pyproject.toml`` describes the package build
+system requirements.
 
-1. ``pip3 install .`` will install these packages to your current
-   environment. If you are inside a virtual environment, they will
-   install there. If you are not, it will attempt to install to the
-   global environment, which is **not recommended**.
+You will generally install these packages by invoking ``pip3 install
+.``; which will install these packages to your current environment. If
+you are inside a virtual environment, they will install there. If you
+are not, modern versions of pip will attempt instead to install to your
+local user environment. Older versions of pip will attempt to install to
+the global environment, which is **not recommended**.
 
-2. ``pip3 install --user .`` will install these packages to your user's
-   local python packages. If you are inside of a virtual environment,
-   this will fail; you want the first invocation above.
-
-If you append the ``--editable`` or ``-e`` argument to either invocation
-above, pip will install in "editable" mode. This installs the package as
-a forwarder ("qemu.egg-link") that points to the source tree. In so
-doing, the installed package always reflects the latest version in your
-source tree.
+If you append the ``--editable`` or ``-e`` argument to the above
+invocation, pip will install in "editable" mode. This installs the
+package as a "forwarder" that points to the source tree, so that the
+installed package always reflects the latest version in your source
+tree.
 
 Installing ".[devel]" instead of "." will additionally pull in required
 packages for testing this package. They are not runtime requirements,
@@ -81,4 +80,4 @@ Files in this directory
 - ``VERSION`` contains the PEP-440 compliant version used to describe
   this package; it is referenced by ``setup.cfg``.
 - ``setup.cfg`` houses setuptools package configuration.
-- ``setup.py`` is the setuptools installer used by pip; See above.
+- ``pyproject.toml`` lists build system requirements for the Python packages.
diff --git a/python/Makefile b/python/Makefile
index 764b79ccb23..845fbb66cf4 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -15,8 +15,8 @@ help:
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
 	@echo "    These tests use the newest dependencies."
-	@echo "    Requires: Python 3.9 - 3.11, and tox."
-	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.11'"
+	@echo "    Requires: Python 3.9 - 3.13, and tox."
+	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.13'"
 	@echo "    The variable QEMU_TOX_EXTRA_ARGS can be use to pass extra"
 	@echo "    arguments to tox".
 	@echo ""
@@ -63,12 +63,12 @@ $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.tx
 	@(								\
 		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
 		. $(QEMU_MINVENV_DIR)/bin/activate;			\
-		echo "INSTALL wheel $(QEMU_MINVENV_DIR)";               \
-		$(PIP_INSTALL) wheel 1>/dev/null;		               \
+		echo "INSTALL wheel $(QEMU_MINVENV_DIR)";		\
+		$(PIP_INSTALL) wheel 1>/dev/null;			\
 		echo "INSTALL -r tests/minreqs.txt $(QEMU_MINVENV_DIR)";\
 		$(PIP_INSTALL) -r tests/minreqs.txt 1>/dev/null;	\
 		echo "INSTALL -e qemu $(QEMU_MINVENV_DIR)";		\
-		$(PIP_INSTALL) -e . 1>/dev/null;			\
+		$(PIP_INSTALL) -e . --config-settings=editable_mode=compat 1>/dev/null;			\
 	)
 	@touch $(QEMU_MINVENV_DIR)
 
@@ -103,7 +103,7 @@ check-dev: dev-venv
 
 .PHONY: develop
 develop:
-	$(PIP_INSTALL) -e .[devel]
+	$(PIP_INSTALL) -e .[devel] --config-settings=editable_mode=compat
 
 .PHONY: check
 check:
@@ -122,12 +122,12 @@ check-coverage:
 
 .PHONY: clean
 clean:
-	python3 setup.py clean --all
-	rm -f pyproject.toml
+	rm -rf build/
 
 .PHONY: distclean
 distclean: clean
-	rm -rf qemu.egg-info/ .eggs/ dist/
+	rm -rf qemu.egg-info/
 	rm -rf $(QEMU_VENV_DIR) $(QEMU_MINVENV_DIR) .tox/
+	rm -rf .mypy_cache/
 	rm -f .coverage .coverage.*
 	rm -rf htmlcov/
diff --git a/python/pyproject.toml b/python/pyproject.toml
new file mode 100644
index 00000000000..2de34bf4729
--- /dev/null
+++ b/python/pyproject.toml
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+[build-system]
+requires = [
+    "setuptools>=39.2",
+    "wheel",
+]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools_scm]
diff --git a/python/setup.py b/python/setup.py
deleted file mode 100755
index c5bc45919a4..00000000000
--- a/python/setup.py
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-"""
-QEMU tooling installer script
-Copyright (c) 2020-2021 John Snow for Red Hat, Inc.
-"""
-
-import setuptools
-from setuptools.command import bdist_egg
-import sys
-import pkg_resources
-
-
-class bdist_egg_guard(bdist_egg.bdist_egg):
-    """
-    Protect against bdist_egg from being executed
-
-    This prevents calling 'setup.py install' directly, as the 'install'
-    CLI option will invoke the deprecated bdist_egg hook. "pip install"
-    calls the more modern bdist_wheel hook, which is what we want.
-    """
-    def run(self):
-        sys.exit(
-            'Installation directly via setup.py is not supported.\n'
-            'Please use `pip install .` instead.'
-        )
-
-
-def main():
-    """
-    QEMU tooling installer
-    """
-
-    # https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
-    pkg_resources.require('setuptools>=39.2')
-
-    setuptools.setup(cmdclass={'bdist_egg': bdist_egg_guard})
-
-
-if __name__ == '__main__':
-    main()
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index cd2e2a81c3d..e9edfa55eb7 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -15,6 +15,7 @@
 # also pin setuptools to version 70 or below. Otherwise, the
 # installation of the QEMU package itself fails, failing to find
 # setuptools.
+# Avocado also appears to require an older setuptools.
 setuptools<=70
 
 # Dependencies for qapidoc/qapi_domain et al
@@ -55,7 +56,6 @@ astroid==2.15.4
 dill==0.2
 lazy-object-proxy==1.4.0
 platformdirs==2.2.0
-toml==0.10.0
 tomlkit==0.10.1
 wrapt==1.14.0
 
-- 
2.48.1
Re: [PATCH v2 01/12] python: convert packages to PEP517/pyproject.toml
Posted by Thomas Huth 5 months ago
On 12/06/2025 22.54, John Snow wrote:
> Newer versions of setuptools increasingly expect that packages are
> defined using the pyproject.toml/PEP517 packaging layout format. With
> 3.9 as our minimum, I believe it's finally appropriate to make the shift
> away from the legacy packaging format.
> 
> Update documentation and dependencies that change as a result of the
> different build/packaging/installation pathways.
> 
> This change has the effect of fixing "make check-dev", which has been
> broken on newer versions of Fedora for a while, now.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/README.rst        | 33 ++++++++++++++++-----------------
>   python/Makefile          | 18 +++++++++---------
>   python/pyproject.toml    | 10 ++++++++++
>   python/setup.py          | 40 ----------------------------------------
>   python/tests/minreqs.txt |  2 +-
>   5 files changed, 36 insertions(+), 67 deletions(-)
>   create mode 100644 python/pyproject.toml
>   delete mode 100755 python/setup.py

Reviewed-by: Thomas Huth <thuth@redhat.com>
Re: [PATCH v2 01/12] python: convert packages to PEP517/pyproject.toml
Posted by John Snow 5 months ago
On Fri, Jun 13, 2025 at 4:36 AM Thomas Huth <thuth@redhat.com> wrote:

> On 12/06/2025 22.54, John Snow wrote:
> > Newer versions of setuptools increasingly expect that packages are
> > defined using the pyproject.toml/PEP517 packaging layout format. With
> > 3.9 as our minimum, I believe it's finally appropriate to make the shift
> > away from the legacy packaging format.
> >
> > Update documentation and dependencies that change as a result of the
> > different build/packaging/installation pathways.
> >
> > This change has the effect of fixing "make check-dev", which has been
> > broken on newer versions of Fedora for a while, now.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >   python/README.rst        | 33 ++++++++++++++++-----------------
> >   python/Makefile          | 18 +++++++++---------
> >   python/pyproject.toml    | 10 ++++++++++
> >   python/setup.py          | 40 ----------------------------------------
> >   python/tests/minreqs.txt |  2 +-
> >   5 files changed, 36 insertions(+), 67 deletions(-)
> >   create mode 100644 python/pyproject.toml
> >   delete mode 100755 python/setup.py
>
> Reviewed-by: Thomas Huth <thuth@redhat.com>


Thanks. If there are no objections, I might stage at least the first few
reviewed patches here so I can get this stitched up sooner than later. The
more disruptive parts I might wait a touch longer on.