Hello!

This series drops the in-tree version of our python-qemu-qmp package
("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
located at https://gitlab.com/qemu-project/python-qemu-qmp.

GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
       (FreeBSD isn't my fault...!)

The major effects of this patch series are:

1. qemu.qmp will be installed from PyPI or vendored packages instead of
   being utilized directly from the qemu.git tree.
2. There are no new python dependencies checked or installed during
   configure.  All test dependencies are installed post-hoc on an
   as-needed basis.
3. "make check-venv" is no longer required to be run manually before any
   test that is integrated with meson; this includes "make check" and
   "make check-functional".
4. "make check-venv" no longer installs functional test dependencies: it
   installs only the core suite of python test dependencies.
5. "make check-venv" is now required as a pre-requisite for running
   device-crash-test and manually executed iotests.
6. Unfortunately, python3-wheel and python3-setuptools are now required
   on the host system if tests are to be executed and >= Python 3.13 is
   used.
7. An awful lot of deleted lines of code, and a lot fewer headaches
   managing two nearly-identical copies of this source code. O:-)

Patches 1-5 are build system focused; they set up new pythondeps.toml,
mkvenv, and meson systems in preparation for relying on an external
qemu.qmp library, but does not yet make the switch.

Patches 6-9 are testing and CI focused; they add necessary preparation
steps to keep tests running happily once the in-tree qemu.qmp library is
removed.

Patches 10-15 are build system focused again; they implement the actual
switchover to the external qemu.qmp library.

v3:

 - Fixed FreeBSD vm test
 - Fixed Cirrus macOS test (Needed setuptools as a dep, "file:///" had
   one too many forward slashes, which works fine for older
   pip/setuptools, but was incompatibly fixed recently.)
 - Fixed an issue that prevented installing the local python packages in
   an offline, isolated build environment (need --no-isolated-build and
   pip environment variable flags)
 - Added the python3-wheel and python3-setuptools build dependencies to
   lcitool, owing to the above issues
 - Ensured that the $download flag (for PyPI access) from configure is
   carried over into pytest/meson.build invocations.  This defaults to
   "enabled", so test dependencies should not have any issue being
   fetched on most developer workstations and build environments.
 - Fixed several bugs in the mkvenv.py local package installation mechanism.
 - Flags and workarounds added for installing local packages offline now
   apply *only* to local package installations, and not "normal"
   depspecs.
 - Reordered patches into more logical "phases" for easier review (I
   hope...?)

Known issues: "make check-build" still installs all of the test
dependencies to the pyvenv. I'm not sure if this is desired or not, but
I ran into problems preventing it from doing this without breaking a ton
of CI tests. I'll continue to investigate this, but wanted to send this
v3 out.

v2:

 - move "make check-venv" earlier in GitLab CI/CD pipeline, to avoid
   re-running configure
 - Fix functional tests not using PyPI to fetch pygdbmi package
 - Remove pre-requisites which are now merged

--js

John Snow (15):
  python/mkvenv: create timestamp file for each group "ensured"
  python/mkvenv: bump 'qemu.qmp' dependency for testdeps
  python/mkvenv: add 'checktests' and 'functests' dependency groups
  python/mkvenv: add mechanism to install local package(s)
  meson, mkvenv: add checktests and functests custom targets
  tests: Use configured python to run GitLab iotests
  tests: run "make check-venv" before running iotests
  tests: ensure "make check-venv" is run for crash tests
  tests/lcitool: add python3 wheel and setuptools deps for qemu
  python: add vendored qemu.qmp package
  meson, mkvenv: make iotests depend on checktests group
  meson, mkvenv: make functional tests depend on functests group
  meson, mkvenv: add qemu.git/python/qemu package to pythondeps.toml
  tests: replace old "check-venv" target with meson target
  python: delete qemu.qmp

 meson.build                                   |    1 +
 .gitlab-ci.d/buildtest.yml                    |   11 +-
 .gitlab-ci.d/cirrus/freebsd-14.vars           |    2 +-
 .gitlab-ci.d/cirrus/macos-14.vars             |    2 +-
 python/qemu/qmp/__init__.py                   |   60 -
 python/qemu/qmp/error.py                      |   53 -
 python/qemu/qmp/events.py                     |  751 -----------
 python/qemu/qmp/legacy.py                     |  339 -----
 python/qemu/qmp/message.py                    |  217 ----
 python/qemu/qmp/models.py                     |  146 ---
 python/qemu/qmp/protocol.py                   | 1101 -----------------
 python/qemu/qmp/py.typed                      |    0
 python/qemu/qmp/qmp_client.py                 |  732 -----------
 python/qemu/qmp/qmp_shell.py                  |  689 -----------
 python/qemu/qmp/qmp_tui.py                    |  665 ----------
 python/qemu/qmp/util.py                       |  150 ---
 python/qemu/utils/qom_fuse.py                 |    1 -
 python/scripts/mkvenv.py                      |   44 +-
 python/scripts/vendor.py                      |    2 +
 python/setup.cfg                              |   31 +-
 python/tests/minreqs.txt                      |    8 +-
 python/tests/protocol.py                      |  596 ---------
 python/wheels/qemu_qmp-0.0.5-py3-none-any.whl |  Bin 0 -> 72263 bytes
 pythondeps.toml                               |   13 +-
 pyvenv/meson.build                            |   40 +
 tests/Makefile.include                        |   25 +-
 tests/docker/dockerfiles/alpine.docker        |    2 +
 tests/docker/dockerfiles/centos9.docker       |    2 +
 tests/functional/meson.build                  |    7 +-
 tests/lcitool/projects/qemu.yml               |    2 +
 tests/qemu-iotests/meson.build                |    2 +-
 tests/vm/generated/freebsd.json               |    2 +
 32 files changed, 126 insertions(+), 5570 deletions(-)
 delete mode 100644 python/qemu/qmp/__init__.py
 delete mode 100644 python/qemu/qmp/error.py
 delete mode 100644 python/qemu/qmp/events.py
 delete mode 100644 python/qemu/qmp/legacy.py
 delete mode 100644 python/qemu/qmp/message.py
 delete mode 100644 python/qemu/qmp/models.py
 delete mode 100644 python/qemu/qmp/protocol.py
 delete mode 100644 python/qemu/qmp/py.typed
 delete mode 100644 python/qemu/qmp/qmp_client.py
 delete mode 100644 python/qemu/qmp/qmp_shell.py
 delete mode 100644 python/qemu/qmp/qmp_tui.py
 delete mode 100644 python/qemu/qmp/util.py
 delete mode 100644 python/tests/protocol.py
 create mode 100644 python/wheels/qemu_qmp-0.0.5-py3-none-any.whl
 create mode 100644 pyvenv/meson.build

-- 
2.51.1

First of all: Ping!

Please-Review: Daniel P. Berrangé <berrange@redhat.com>
Please-Review: Paolo Bonzini <pbonzini@redhat.com>
Please-Review: Alex Bennée <alex.bennee@linaro.org>

It would be nice to get this series staged right as the tree opens so
that I can address any build issues it creates in a timely manner
during the next window.

Second, [below, in-line] ...

On Fri, Dec 5, 2025 at 1:01 AM John Snow <jsnow@redhat.com> wrote:
>
> Hello!
>
> This series drops the in-tree version of our python-qemu-qmp package
> ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> located at https://gitlab.com/qemu-project/python-qemu-qmp.
>
> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
>        (FreeBSD isn't my fault...!)
>
> The major effects of this patch series are:
>
> 1. qemu.qmp will be installed from PyPI or vendored packages instead of
>    being utilized directly from the qemu.git tree.
> 2. There are no new python dependencies checked or installed during
>    configure.  All test dependencies are installed post-hoc on an
>    as-needed basis.
> 3. "make check-venv" is no longer required to be run manually before any
>    test that is integrated with meson; this includes "make check" and
>    "make check-functional".
> 4. "make check-venv" no longer installs functional test dependencies: it
>    installs only the core suite of python test dependencies.
> 5. "make check-venv" is now required as a pre-requisite for running
>    device-crash-test and manually executed iotests.
> 6. Unfortunately, python3-wheel and python3-setuptools are now required
>    on the host system if tests are to be executed and >= Python 3.13 is
>    used.
> 7. An awful lot of deleted lines of code, and a lot fewer headaches
>    managing two nearly-identical copies of this source code. O:-)
>
> Patches 1-5 are build system focused; they set up new pythondeps.toml,
> mkvenv, and meson systems in preparation for relying on an external
> qemu.qmp library, but does not yet make the switch.
>
> Patches 6-9 are testing and CI focused; they add necessary preparation
> steps to keep tests running happily once the in-tree qemu.qmp library is
> removed.
>
> Patches 10-15 are build system focused again; they implement the actual
> switchover to the external qemu.qmp library.
>
> v3:
>
>  - Fixed FreeBSD vm test
>  - Fixed Cirrus macOS test (Needed setuptools as a dep, "file:///" had
>    one too many forward slashes, which works fine for older
>    pip/setuptools, but was incompatibly fixed recently.)
>  - Fixed an issue that prevented installing the local python packages in
>    an offline, isolated build environment (need --no-isolated-build and
>    pip environment variable flags)
>  - Added the python3-wheel and python3-setuptools build dependencies to
>    lcitool, owing to the above issues
>  - Ensured that the $download flag (for PyPI access) from configure is
>    carried over into pytest/meson.build invocations.  This defaults to
>    "enabled", so test dependencies should not have any issue being
>    fetched on most developer workstations and build environments.
>  - Fixed several bugs in the mkvenv.py local package installation mechanism.
>  - Flags and workarounds added for installing local packages offline now
>    apply *only* to local package installations, and not "normal"
>    depspecs.
>  - Reordered patches into more logical "phases" for easier review (I
>    hope...?)
>
> Known issues: "make check-build" still installs all of the test
> dependencies to the pyvenv. I'm not sure if this is desired or not, but
> I ran into problems preventing it from doing this without breaking a ton
> of CI tests. I'll continue to investigate this, but wanted to send this
> v3 out.

I sussed it out. I *can* remove the pyvenv targets from "check-build",
and the reason CI tests fail when I do this is because the build
templates use NINJA=":", so running "make check" suppresses the meson
dependencies. It can be rectified by setting eight of the twelve total
occurrences of "make check-build" in our CI files to also run "make
check-venv".

It's not clear if we want this or not. It depends on what
"check-build" semantically means. On the one hand, it doesn't do any
prefetch for the functional tests, so you could make the argument that
environment setup is similar. On the other hand, I think it's used
almost exclusively by and for the CI test suite, in which case it
running the environment setup is likely a pretty helpful/useful thing.
It's something Thomas pointed out, but I don't feel like I have a
strong opinion on it currently.

--js

>
> v2:
>
>  - move "make check-venv" earlier in GitLab CI/CD pipeline, to avoid
>    re-running configure
>  - Fix functional tests not using PyPI to fetch pygdbmi package
>  - Remove pre-requisites which are now merged
>
> --js
>
> John Snow (15):
>   python/mkvenv: create timestamp file for each group "ensured"
>   python/mkvenv: bump 'qemu.qmp' dependency for testdeps
>   python/mkvenv: add 'checktests' and 'functests' dependency groups
>   python/mkvenv: add mechanism to install local package(s)
>   meson, mkvenv: add checktests and functests custom targets
>   tests: Use configured python to run GitLab iotests
>   tests: run "make check-venv" before running iotests
>   tests: ensure "make check-venv" is run for crash tests
>   tests/lcitool: add python3 wheel and setuptools deps for qemu
>   python: add vendored qemu.qmp package
>   meson, mkvenv: make iotests depend on checktests group
>   meson, mkvenv: make functional tests depend on functests group
>   meson, mkvenv: add qemu.git/python/qemu package to pythondeps.toml
>   tests: replace old "check-venv" target with meson target
>   python: delete qemu.qmp
>
>  meson.build                                   |    1 +
>  .gitlab-ci.d/buildtest.yml                    |   11 +-
>  .gitlab-ci.d/cirrus/freebsd-14.vars           |    2 +-
>  .gitlab-ci.d/cirrus/macos-14.vars             |    2 +-
>  python/qemu/qmp/__init__.py                   |   60 -
>  python/qemu/qmp/error.py                      |   53 -
>  python/qemu/qmp/events.py                     |  751 -----------
>  python/qemu/qmp/legacy.py                     |  339 -----
>  python/qemu/qmp/message.py                    |  217 ----
>  python/qemu/qmp/models.py                     |  146 ---
>  python/qemu/qmp/protocol.py                   | 1101 -----------------
>  python/qemu/qmp/py.typed                      |    0
>  python/qemu/qmp/qmp_client.py                 |  732 -----------
>  python/qemu/qmp/qmp_shell.py                  |  689 -----------
>  python/qemu/qmp/qmp_tui.py                    |  665 ----------
>  python/qemu/qmp/util.py                       |  150 ---
>  python/qemu/utils/qom_fuse.py                 |    1 -
>  python/scripts/mkvenv.py                      |   44 +-
>  python/scripts/vendor.py                      |    2 +
>  python/setup.cfg                              |   31 +-
>  python/tests/minreqs.txt                      |    8 +-
>  python/tests/protocol.py                      |  596 ---------
>  python/wheels/qemu_qmp-0.0.5-py3-none-any.whl |  Bin 0 -> 72263 bytes
>  pythondeps.toml                               |   13 +-
>  pyvenv/meson.build                            |   40 +
>  tests/Makefile.include                        |   25 +-
>  tests/docker/dockerfiles/alpine.docker        |    2 +
>  tests/docker/dockerfiles/centos9.docker       |    2 +
>  tests/functional/meson.build                  |    7 +-
>  tests/lcitool/projects/qemu.yml               |    2 +
>  tests/qemu-iotests/meson.build                |    2 +-
>  tests/vm/generated/freebsd.json               |    2 +
>  32 files changed, 126 insertions(+), 5570 deletions(-)
>  delete mode 100644 python/qemu/qmp/__init__.py
>  delete mode 100644 python/qemu/qmp/error.py
>  delete mode 100644 python/qemu/qmp/events.py
>  delete mode 100644 python/qemu/qmp/legacy.py
>  delete mode 100644 python/qemu/qmp/message.py
>  delete mode 100644 python/qemu/qmp/models.py
>  delete mode 100644 python/qemu/qmp/protocol.py
>  delete mode 100644 python/qemu/qmp/py.typed
>  delete mode 100644 python/qemu/qmp/qmp_client.py
>  delete mode 100644 python/qemu/qmp/qmp_shell.py
>  delete mode 100644 python/qemu/qmp/qmp_tui.py
>  delete mode 100644 python/qemu/qmp/util.py
>  delete mode 100644 python/tests/protocol.py
>  create mode 100644 python/wheels/qemu_qmp-0.0.5-py3-none-any.whl
>  create mode 100644 pyvenv/meson.build
>
> --
> 2.51.1
>
>

On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
> Hello!
> 
> This series drops the in-tree version of our python-qemu-qmp package
> ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> located at https://gitlab.com/qemu-project/python-qemu-qmp.
> 
> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
>        (FreeBSD isn't my fault...!)
> 
> The major effects of this patch series are:
> 
> 1. qemu.qmp will be installed from PyPI or vendored packages instead of
>    being utilized directly from the qemu.git tree.

This is not getting installed in enough scenarios IMHO.

My current workflow is commonly

  $ ./configure --target-list=x86_64-softmmu
  $ make
  $ ./scripts/qmp/qmp-shell-wrap 
   /var/home/berrange/src/virt/qemu/build
  Traceback (most recent call last):
    File "/var/home/berrange/src/virt/qemu/scripts/qmp/qmp-shell-wrap", line 7, in <module>
      from qemu.qmp import qmp_shell
  ModuleNotFoundError: No module named 'qemu.qmp'

Even if I add in a call to 'source build/pyvenv/bin/activate'
after 'make', I'm still missing the qemu.qmp python module.

AFAICT, qemu.qmp only gets installed in the venv if you run
'make check', and IMHO that should not be expected for the
above usage scenario.

Likewise I can no longer run any of the test programs directly
without first having run 'make check'. eg what I would currently
do is:

   $ ./configure --target-list=x86_64-softmmu
   $ make
   $ cd build/tests/qemu-iotests
   $ ./check 300

I don't generally run 'make check' as frequently as I
'configure && make' as it adds alot of time which is
not needed usually.

In general I find it tedious having to remember to run
'source build/pyvenv/bin/activate' in every terminal
window where I might want to run a QEMU script that relies
on python. It is also fairly undesirable too, as I use the
same terminal window for various purposes, that activating
the QEMU venv can interfere with non-QEMU python usage.

I would find it helpful if we provided a "run" script in the
root dir of the build directory that contained the following

  $ cat build/run
  #!/bin/sh

  set -e

  HERE=$(realpath $(dirname $0))
  source $HERE/pyvenv/bin/activate
  exec "$@"

Which would be used as a wrapper to execute scripts with the
right environment. That would let us keep the simpler workflow,
and avoid polluting the global terminal environment with the
qemu venv.

eg to be used as

   $ ./configure --target-list=x86_64-softmmu
   $ make
   $ cd build/tests/qemu-iotests
   $ ../../run ./check 300

or

  $ ./configure --target-list=x86_64-softmmu
  $ make
  $ ./build/run ./scripts/qmp/qmp-shell-wrap 



> 2. There are no new python dependencies checked or installed during
>    configure.  All test dependencies are installed post-hoc on an
>    as-needed basis.
> 3. "make check-venv" is no longer required to be run manually before any
>    test that is integrated with meson; this includes "make check" and
>    "make check-functional".
> 4. "make check-venv" no longer installs functional test dependencies: it
>    installs only the core suite of python test dependencies.
> 5. "make check-venv" is now required as a pre-requisite for running
>    device-crash-test and manually executed iotests.
> 6. Unfortunately, python3-wheel and python3-setuptools are now required
>    on the host system if tests are to be executed and >= Python 3.13 is
>    used.
> 7. An awful lot of deleted lines of code, and a lot fewer headaches
>    managing two nearly-identical copies of this source code. O:-)
> 
> Patches 1-5 are build system focused; they set up new pythondeps.toml,
> mkvenv, and meson systems in preparation for relying on an external
> qemu.qmp library, but does not yet make the switch.
> 
> Patches 6-9 are testing and CI focused; they add necessary preparation
> steps to keep tests running happily once the in-tree qemu.qmp library is
> removed.
> 
> Patches 10-15 are build system focused again; they implement the actual
> switchover to the external qemu.qmp library.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
> > Hello!
> >
> > This series drops the in-tree version of our python-qemu-qmp package
> > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> > located at https://gitlab.com/qemu-project/python-qemu-qmp.
> >
> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
> >        (FreeBSD isn't my fault...!)
> >
> > The major effects of this patch series are:
> >
> > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
> >    being utilized directly from the qemu.git tree.
>
> This is not getting installed in enough scenarios IMHO.
>

It's hard to trigger an install when you don't use the build system,
though... If you bypass make/meson/ninja entirely I'm not really sure what
I can do to bootstrap the test deps.


> My current workflow is commonly
>
>   $ ./configure --target-list=x86_64-softmmu
>   $ make
>   $ ./scripts/qmp/qmp-shell-wrap
>    /var/home/berrange/src/virt/qemu/build
>   Traceback (most recent call last):
>     File "/var/home/berrange/src/virt/qemu/scripts/qmp/qmp-shell-wrap",
> line 7, in <module>
>       from qemu.qmp import qmp_shell
>   ModuleNotFoundError: No module named 'qemu.qmp'
>
> Even if I add in a call to 'source build/pyvenv/bin/activate'
> after 'make', I'm still missing the qemu.qmp python module.
>
> AFAICT, qemu.qmp only gets installed in the venv if you run
> 'make check', and IMHO that should not be expected for the
> above usage scenario.
>

"make check-venv", to do just the minimum setup. You don't have to run
check.

We could always add a flag to configure to front-load the testing
environment and scripts, but I thought it was best to avoid adding new
dependencies by default.

Like --with-python-tests or something? Then you'd always have it like you
do with meson and sphinx (with --enable-docs)


> Likewise I can no longer run any of the test programs directly
> without first having run 'make check'. eg what I would currently
> do is:
>
>    $ ./configure --target-list=x86_64-softmmu
>    $ make
>    $ cd build/tests/qemu-iotests
>    $ ./check 300
>
> I don't generally run 'make check' as frequently as I
> 'configure && make' as it adds alot of time which is
> not needed usually.
>
> In general I find it tedious having to remember to run
> 'source build/pyvenv/bin/activate' in every terminal
> window where I might want to run a QEMU script that relies
> on python. It is also fairly undesirable too, as I use the
> same terminal window for various purposes, that activating
> the QEMU venv can interfere with non-QEMU python usage.
>

You can run one-offs with build/pyvenv/bin/python3 without activating the
environment,

but you're right that this does require an environment setup step (make
check-venv)


> I would find it helpful if we provided a "run" script in the
> root dir of the build directory that contained the following
>
>   $ cat build/run
>   #!/bin/sh
>
>   set -e
>
>   HERE=$(realpath $(dirname $0))
>   source $HERE/pyvenv/bin/activate
>   exec "$@"
>
> Which would be used as a wrapper to execute scripts with the
> right environment. That would let us keep the simpler workflow,
> and avoid polluting the global terminal environment with the
> qemu venv.
>
> eg to be used as
>
>    $ ./configure --target-list=x86_64-softmmu
>    $ make
>    $ cd build/tests/qemu-iotests
>    $ ../../run ./check 300
>
> or
>
>   $ ./configure --target-list=x86_64-softmmu
>   $ make
>   $ ./build/run ./scripts/qmp/qmp-shell-wrap


Shouldn't be too hard to add, honestly. Maybe adding in a "meson compile
pyvenv/pyvenv_tests_group" line at the start to trigger the dep
installation if it hasn't occurred already?

Or maybe just directly invoking mkvenv ensuregroup to avoid ninja
re-configuring things while you develop.

Something like that.



>
>
>
> > 2. There are no new python dependencies checked or installed during
> >    configure.  All test dependencies are installed post-hoc on an
> >    as-needed basis.
> > 3. "make check-venv" is no longer required to be run manually before any
> >    test that is integrated with meson; this includes "make check" and
> >    "make check-functional".
> > 4. "make check-venv" no longer installs functional test dependencies: it
> >    installs only the core suite of python test dependencies.
> > 5. "make check-venv" is now required as a pre-requisite for running
> >    device-crash-test and manually executed iotests.
> > 6. Unfortunately, python3-wheel and python3-setuptools are now required
> >    on the host system if tests are to be executed and >= Python 3.13 is
> >    used.
> > 7. An awful lot of deleted lines of code, and a lot fewer headaches
> >    managing two nearly-identical copies of this source code. O:-)
> >
> > Patches 1-5 are build system focused; they set up new pythondeps.toml,
> > mkvenv, and meson systems in preparation for relying on an external
> > qemu.qmp library, but does not yet make the switch.
> >
> > Patches 6-9 are testing and CI focused; they add necessary preparation
> > steps to keep tests running happily once the in-tree qemu.qmp library is
> > removed.
> >
> > Patches 10-15 are build system focused again; they implement the actual
> > switchover to the external qemu.qmp library.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

On Tue, Dec 09, 2025 at 12:23:21PM -0500, John Snow wrote:
> On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
> > > Hello!
> > >
> > > This series drops the in-tree version of our python-qemu-qmp package
> > > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> > > located at https://gitlab.com/qemu-project/python-qemu-qmp.
> > >
> > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
> > >        (FreeBSD isn't my fault...!)
> > >
> > > The major effects of this patch series are:
> > >
> > > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
> > >    being utilized directly from the qemu.git tree.
> >
> > This is not getting installed in enough scenarios IMHO.
> >
> 
> It's hard to trigger an install when you don't use the build system,

Well I am using the build system in the same way that I've always
used it with QEMU. IOW, the benchmark for this is the current way
things work with the python stuff in tree. The ideal would be to
match that behaviour with no workflow changes needed, if it is
practical to achieve that without major downsides. 

> though... If you bypass make/meson/ninja entirely I'm not really sure what
> I can do to bootstrap the test deps.

Can we just do it unconditionally in configure / meson ? These aren't
big files to download, and we're already dealing other python stuff
for the build, and git submodules, and rust crates. Pulling in qemu.qmp
doesn't feel like it is a burden to do by default ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Dec 9, 2025 at 2:46 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Dec 09, 2025 at 12:23:21PM -0500, John Snow wrote:
> > On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <berrange@redhat.com>
> > wrote:
> >
> > > On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
> > > > Hello!
> > > >
> > > > This series drops the in-tree version of our python-qemu-qmp package
> > > > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> > > > located at https://gitlab.com/qemu-project/python-qemu-qmp.
> > > >
> > > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
> > > >        (FreeBSD isn't my fault...!)
> > > >
> > > > The major effects of this patch series are:
> > > >
> > > > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
> > > >    being utilized directly from the qemu.git tree.
> > >
> > > This is not getting installed in enough scenarios IMHO.
> > >
> >
> > It's hard to trigger an install when you don't use the build system,
>
> Well I am using the build system in the same way that I've always
> used it with QEMU. IOW, the benchmark for this is the current way
> things work with the python stuff in tree. The ideal would be to
> match that behaviour with no workflow changes needed, if it is
> practical to achieve that without major downsides.

I know, but if we add out-of-tree things, there's some fundamental
things that change - we need to load that dependency somewhere,
somewhen.

>
> > though... If you bypass make/meson/ninja entirely I'm not really sure what
> > I can do to bootstrap the test deps.
>
> Can we just do it unconditionally in configure / meson ? These aren't
> big files to download, and we're already dealing other python stuff
> for the build, and git submodules, and rust crates. Pulling in qemu.qmp
> doesn't feel like it is a burden to do by default ?

I definitely could, and I think it would be rather convenient; I only
have some minor concerns about it:

- We don't promise tests on all platforms that we promise builds on.
We have definitely broken this in the past. It might work currently,
but it's a line I want to be aware we are crossing. It may necessitate
python3-wheel and python3-setuptools just to build in this case.
That's probably not an issue on workstations, but it's more bricks on
the wall that are actually not truly needed until you run tests (or
prepare to run tests).

- Configure will get slower by default. I can install the core test
deps by default if people don't mind the additional delay time. It
might be something like 2-4 seconds, depending. If you don't care, I
don't.

- Things like functional tests are still going to require some kind of
environment prep for all the extra stuff they require, I don't think
it's reasonable to preload all of that stuff at configure time, and we
never have. "make check-functional" is sufficient to pull those deps
in, but if there are ways to engage the functional tests manually that
people are using, I think another preparation step is unavoidable
there.

So, in addition to the "pyrun" wrapper I mailed in a separate reply
(which I think is a good idea anyway, regardless of what direction we
go here), I see two main paths that address your issues in differing
amounts:

(1) make configure prepare the test deps *by default*, adding a flag
like --without-test-deps to disable it, or

(2) Continue not prepping the test deps, but allow --with-test-deps to
pre-load them.

More or less the same solution, just with different defaults. I'm
fairly ambivalent, my only personal "habit" here is "I am really
reluctant to touch configure unless there is a strong consensus around
it because I dislike having to make arguments at that level."

>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

On Tue, Dec 09, 2025 at 03:05:43PM -0500, John Snow wrote:
> On Tue, Dec 9, 2025 at 2:46 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Dec 09, 2025 at 12:23:21PM -0500, John Snow wrote:
> > > On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <berrange@redhat.com>
> > > wrote:
> > >
> > > > On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
> > > > > Hello!
> > > > >
> > > > > This series drops the in-tree version of our python-qemu-qmp package
> > > > > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> > > > > located at https://gitlab.com/qemu-project/python-qemu-qmp.
> > > > >
> > > > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
> > > > >        (FreeBSD isn't my fault...!)
> > > > >
> > > > > The major effects of this patch series are:
> > > > >
> > > > > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
> > > > >    being utilized directly from the qemu.git tree.
> > > >
> > > > This is not getting installed in enough scenarios IMHO.
> > > >
> > >
> > > It's hard to trigger an install when you don't use the build system,
> >
> > Well I am using the build system in the same way that I've always
> > used it with QEMU. IOW, the benchmark for this is the current way
> > things work with the python stuff in tree. The ideal would be to
> > match that behaviour with no workflow changes needed, if it is
> > practical to achieve that without major downsides.
> 
> I know, but if we add out-of-tree things, there's some fundamental
> things that change - we need to load that dependency somewhere,
> somewhen.

Yep, there's a tradeoff, so we have to figure out what's practical
to achieve.

> > > though... If you bypass make/meson/ninja entirely I'm not really sure what
> > > I can do to bootstrap the test deps.
> >
> > Can we just do it unconditionally in configure / meson ? These aren't
> > big files to download, and we're already dealing other python stuff
> > for the build, and git submodules, and rust crates. Pulling in qemu.qmp
> > doesn't feel like it is a burden to do by default ?
> 
> I definitely could, and I think it would be rather convenient; I only
> have some minor concerns about it:
> 
> - We don't promise tests on all platforms that we promise builds on.
> We have definitely broken this in the past. It might work currently,
> but it's a line I want to be aware we are crossing. It may necessitate
> python3-wheel and python3-setuptools just to build in this case.
> That's probably not an issue on workstations, but it's more bricks on
> the wall that are actually not truly needed until you run tests (or
> prepare to run tests).

We should focus on doing what's optimal for platforms where
tests do work, since they're inherantly our most important
platforms. If we proactively fetch python deps for tests that
happen to be broken on a platform already, I'm not too fussed
about that, as long as we don't break things more than they
are already broken.


> - Configure will get slower by default. I can install the core test
> deps by default if people don't mind the additional delay time. It
> might be something like 2-4 seconds, depending. If you don't care, I
> don't.

IIUC,  'pip' will have a local cache of downloads somewhere under
$HOME, so presumably when we install deps, we're only going to have
a download hit the very first time on a machine, or when the needed
versions change ? If we're mostly just copying files out of the
local cache, the time overhead should remain fairly minimal most of
the time.

We have precedent for downloading stuff at configure stage from
the Rust deps and git submodules. Those Rust deps are used for
built code rather than tests, but if we did need some rust deps
only for tests, presumably they'd still be done in configure ?

> - Things like functional tests are still going to require some kind of
> environment prep for all the extra stuff they require, I don't think
> it's reasonable to preload all of that stuff at configure time, and we
> never have. "make check-functional" is sufficient to pull those deps
> in, but if there are ways to engage the functional tests manually that
> people are using, I think another preparation step is unavoidable
> there.

I'd we interested to know how much overhead loading the functional
test deps actually is ?

As a conceptual point beyond this series, I've always tended to expect
that a "configure" script (or equivalent) is responsible for validating
the host OS pre-requisites and/or fetching neede pre-requisites.

Anything after that (whether builds or tests) would then (mostly) work
from stuff that was already acquired by configure.

This lets devs see upfront if there are any problems with their host,
instead of the host problems being hit later where the errors are
mixed in with build logs.

With functional tests we of course don't want to download the assets
upfront as that's GB's of data. I would tend towards suggesting that
any python deps should be fetched upfront at the configure stage
though, unless we see an unacceptable time hit from that.

> So, in addition to the "pyrun" wrapper I mailed in a separate reply
> (which I think is a good idea anyway, regardless of what direction we
> go here), I see two main paths that address your issues in differing
> amounts:

I sent that as a formal proposal here:

  https://lists.nongnu.org/archive/html/qemu-devel/2025-12/msg01351.html

For the functional test example illustrated there though, we would
preferrably want the functional test deps fetched upfront.

> (1) make configure prepare the test deps *by default*, adding a flag
> like --without-test-deps to disable it, or

Note, this is not just about test deps. The qemu python code is used by
other misc QEMU developer tools, such as the qmp-shell command. I'm
considering the python deps as more like "QEMU build env pre-requisites",
which is what pushed me towards believeing they should be fetched by
default

> (2) Continue not prepping the test deps, but allow --with-test-deps to
> pre-load them.
> 
> More or less the same solution, just with different defaults. I'm
> fairly ambivalent, my only personal "habit" here is "I am really
> reluctant to touch configure unless there is a strong consensus around
> it because I dislike having to make arguments at that level."


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Dec 9, 2025 at 12:23 PM John Snow <jsnow@redhat.com> wrote:
>
>
>
> On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>>
>> On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
>> > Hello!
>> >
>> > This series drops the in-tree version of our python-qemu-qmp package
>> > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
>> > located at https://gitlab.com/qemu-project/python-qemu-qmp.
>> >
>> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
>> >        (FreeBSD isn't my fault...!)
>> >
>> > The major effects of this patch series are:
>> >
>> > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
>> >    being utilized directly from the qemu.git tree.
>>
>> This is not getting installed in enough scenarios IMHO.
>
>
> It's hard to trigger an install when you don't use the build system, though... If you bypass make/meson/ninja entirely I'm not really sure what I can do to bootstrap the test deps.
>
>>
>> My current workflow is commonly
>>
>>   $ ./configure --target-list=x86_64-softmmu
>>   $ make
>>   $ ./scripts/qmp/qmp-shell-wrap
>>    /var/home/berrange/src/virt/qemu/build
>>   Traceback (most recent call last):
>>     File "/var/home/berrange/src/virt/qemu/scripts/qmp/qmp-shell-wrap", line 7, in <module>
>>       from qemu.qmp import qmp_shell
>>   ModuleNotFoundError: No module named 'qemu.qmp'
>>
>> Even if I add in a call to 'source build/pyvenv/bin/activate'
>> after 'make', I'm still missing the qemu.qmp python module.
>>
>> AFAICT, qemu.qmp only gets installed in the venv if you run
>> 'make check', and IMHO that should not be expected for the
>> above usage scenario.
>
>
> "make check-venv", to do just the minimum setup. You don't have to run check.
>
> We could always add a flag to configure to front-load the testing environment and scripts, but I thought it was best to avoid adding new dependencies by default.
>
> Like --with-python-tests or something? Then you'd always have it like you do with meson and sphinx (with --enable-docs)
>
>>
>> Likewise I can no longer run any of the test programs directly
>> without first having run 'make check'. eg what I would currently
>> do is:
>>
>>    $ ./configure --target-list=x86_64-softmmu
>>    $ make
>>    $ cd build/tests/qemu-iotests
>>    $ ./check 300
>>
>> I don't generally run 'make check' as frequently as I
>> 'configure && make' as it adds alot of time which is
>> not needed usually.
>>
>> In general I find it tedious having to remember to run
>> 'source build/pyvenv/bin/activate' in every terminal
>> window where I might want to run a QEMU script that relies
>> on python. It is also fairly undesirable too, as I use the
>> same terminal window for various purposes, that activating
>> the QEMU venv can interfere with non-QEMU python usage.
>
>
> You can run one-offs with build/pyvenv/bin/python3 without activating the environment,
>
> but you're right that this does require an environment setup step (make check-venv)
>
>>
>> I would find it helpful if we provided a "run" script in the
>> root dir of the build directory that contained the following
>>
>>   $ cat build/run
>>   #!/bin/sh
>>
>>   set -e
>>
>>   HERE=$(realpath $(dirname $0))
>>   source $HERE/pyvenv/bin/activate
>>   exec "$@"
>>
>> Which would be used as a wrapper to execute scripts with the
>> right environment. That would let us keep the simpler workflow,
>> and avoid polluting the global terminal environment with the
>> qemu venv.
>>
>> eg to be used as
>>
>>    $ ./configure --target-list=x86_64-softmmu
>>    $ make
>>    $ cd build/tests/qemu-iotests
>>    $ ../../run ./check 300
>>
>> or
>>
>>   $ ./configure --target-list=x86_64-softmmu
>>   $ make
>>   $ ./build/run ./scripts/qmp/qmp-shell-wrap
>
>
> Shouldn't be too hard to add, honestly. Maybe adding in a "meson compile pyvenv/pyvenv_tests_group" line at the start to trigger the dep installation if it hasn't occurred already?
>
> Or maybe just directly invoking mkvenv ensuregroup to avoid ninja re-configuring things while you develop.
>
> Something like that.

How about something like this inside of your build root:

```
#!/usr/bin/env python3

import os
from pathlib import Path
import shlex
import subprocess
import sys

def main():
    venvpath = Path(__file__).parent.joinpath('pyvenv')

    groupfile = venvpath.joinpath('checktests.group')
    if not groupfile.exists():
        subprocess.run(["ninja", "pyvenv/checktests.group"])

    env = os.environ.copy()
    env['VIRTUAL_ENV'] = str(venvpath)
    env['PATH'] = str(venvpath.joinpath('bin')) + ':' + env['PATH']
    env['VIRTUAL_ENV_PROMPT'] = 'pyvenv'
    if 'PYTHONHOME' in env:
        del env['PYTHONHOME']

    command = shlex.join(sys.argv[1:])
    res = subprocess.run(command, env=env, shell=True)
    sys.exit(res.returncode)

if __name__ == '__main__':
    main()
```

Then you can run arbitrary things, like "./pyrun which python3" or
whatever you fancy. It only bothers calling ninja if the timestamp
file is not there to help save some startup time and avoid unnecessary
reconfigures. This should work for qmp-shell, qmp-shell-wrap,
qemu-ga-client, qmp-tui, qom, qom-fuse, qom-get, qom-list, qom-set,
qom-tree, device-crash-test, iotests, etc.

>
>
>>
>>
>>
>>
>> > 2. There are no new python dependencies checked or installed during
>> >    configure.  All test dependencies are installed post-hoc on an
>> >    as-needed basis.
>> > 3. "make check-venv" is no longer required to be run manually before any
>> >    test that is integrated with meson; this includes "make check" and
>> >    "make check-functional".
>> > 4. "make check-venv" no longer installs functional test dependencies: it
>> >    installs only the core suite of python test dependencies.
>> > 5. "make check-venv" is now required as a pre-requisite for running
>> >    device-crash-test and manually executed iotests.
>> > 6. Unfortunately, python3-wheel and python3-setuptools are now required
>> >    on the host system if tests are to be executed and >= Python 3.13 is
>> >    used.
>> > 7. An awful lot of deleted lines of code, and a lot fewer headaches
>> >    managing two nearly-identical copies of this source code. O:-)
>> >
>> > Patches 1-5 are build system focused; they set up new pythondeps.toml,
>> > mkvenv, and meson systems in preparation for relying on an external
>> > qemu.qmp library, but does not yet make the switch.
>> >
>> > Patches 6-9 are testing and CI focused; they add necessary preparation
>> > steps to keep tests running happily once the in-tree qemu.qmp library is
>> > removed.
>> >
>> > Patches 10-15 are build system focused again; they implement the actual
>> > switchover to the external qemu.qmp library.
>>
>> With regards,
>> Daniel
>> --
>> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>>