[PATCH 0/9] Use known good releases when installing in pyvenv

Paolo Bonzini posted 9 patches 9 months, 2 weeks ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20230809092933.761524-1-pbonzini@redhat.com
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>, Ed Maste <emaste@freebsd.org>, Li-Wen Hsu <lwhsu@freebsd.org>, Paolo Bonzini <pbonzini@redhat.com>, Ani Sinha <anisinha@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, John Snow <jsnow@redhat.com>, Cleber Rosa <crosa@redhat.com>
.gitlab-ci.d/buildtest.yml                    |   6 +-
.gitlab-ci.d/cirrus/freebsd-13.vars           |   2 +-
.gitlab-ci.d/cirrus/macos-12.vars             |   2 +-
configure                                     |  22 +-
docs/devel/acpi-bits.rst                      |   6 +-
docs/devel/testing.rst                        |  14 +-
python/scripts/mkvenv.py                      | 201 ++++++++++++++++--
python/scripts/vendor.py                      |   5 +-
python/setup.cfg                              |   9 +
python/wheels/tomli-2.0.1-py3-none-any.whl    | Bin 0 -> 12757 bytes
pythondeps.toml                               |  32 +++
.../org.centos/stream/8/x86_64/test-avocado   |   4 +-
scripts/device-crash-test                     |   2 +-
tests/Makefile.include                        |  19 +-
tests/docker/dockerfiles/centos8.docker       |   3 +-
.../dockerfiles/debian-all-test-cross.docker  |   7 +-
.../dockerfiles/debian-amd64-cross.docker     |   6 +-
tests/docker/dockerfiles/debian-amd64.docker  |   4 +
.../dockerfiles/debian-arm64-cross.docker     |   6 +-
.../dockerfiles/debian-armel-cross.docker     |   6 +-
.../dockerfiles/debian-armhf-cross.docker     |   6 +-
.../dockerfiles/debian-hexagon-cross.docker   |   6 +-
.../dockerfiles/debian-mips64el-cross.docker  |   6 +-
.../dockerfiles/debian-mipsel-cross.docker    |   6 +-
.../dockerfiles/debian-ppc64el-cross.docker   |   6 +-
.../dockerfiles/debian-riscv64-cross.docker   |   2 +-
.../dockerfiles/debian-s390x-cross.docker     |   6 +-
.../dockerfiles/debian-tricore-cross.docker   |   2 +
.../dockerfiles/fedora-i386-cross.docker      |   1 +
.../dockerfiles/fedora-win32-cross.docker     |   2 +-
.../dockerfiles/fedora-win64-cross.docker     |   2 +-
tests/docker/dockerfiles/opensuse-leap.docker |  23 +-
tests/docker/dockerfiles/ubuntu2004.docker    |   4 +-
tests/docker/dockerfiles/ubuntu2204.docker    |   1 +
tests/lcitool/libvirt-ci                      |   2 +-
tests/lcitool/mappings.yml                    |  27 ++-
tests/lcitool/projects/qemu.yml               |   3 +-
tests/lcitool/targets/opensuse-leap-15.yml    |   4 +-
tests/requirements.txt                        |   6 -
tests/vm/Makefile.include                     |   2 +-
tests/vm/generated/freebsd.json               |   1 +
41 files changed, 371 insertions(+), 103 deletions(-)
create mode 100644 python/wheels/tomli-2.0.1-py3-none-any.whl
create mode 100644 pythondeps.toml
delete mode 100644 tests/requirements.txt
[PATCH 0/9] Use known good releases when installing in pyvenv
Posted by Paolo Bonzini 9 months, 2 weeks ago
This series introduce a new installation command for mkvenv.py that
retrieves the packages to be installed from a TOML file. This allows
being more flexible in using the system version of a package, while at
the same time using a known-good version when installing the package.
This is important for packages that sometimes have backwards-incompatible
changes or that depend on specific versions of their dependencies.
    
For example, in the case of sphinx we can always use the last version
that supports older versions of Python and especially docutils (Ubuntu
20.04 for example would not support sphinx 6.0), and in the case of
Avocado we can avoid installing a package that conflicts with any
plugins already existing in the system package path.  This in turn
enables using the same virtual environment for both the build and
qemu-iotests.  John has patches for this.

For the configuration file, TOML was chosen because it was more human
readable and easier to edit than JSON.  A parser is available in the
Python 3.11 standard library; for older versions, the API-compatible
replacement tomli is very small (12k).  I am introducing it as a vendored
.whl file because it is not installed by default in most distros (unlike
pip and setuptools which were introduced in 8.0) and because Debian
11 only has it in bullseye-backports.  However, if preferred the patch
"python: use vendored tomli" can be dropped.  In that case, tomli will
have to be installed from either PyPI or bullseye-backports.

While tomli is bundled with pip, this is only true of recent versions
of pip.  Of all the supported OSes pretty much only FreeBSD has a recent
enough version of pip while staying on Python <3.11.  So we cannot use
the same trick that is in place for distlib.

In order to pick the tomli mapping, lcitool is updated to a recent
version.  As a side effect this updates from LEAP 15.4 to 15.5.

Paolo
    

Paolo Bonzini (9):
  python: mkvenv: tweak the matching of --diagnose to depspecs
  python: mkvenv: introduce TOML-like representation of dependencies
  python: mkvenv: add ensuregroup command
  lcitool: bump libvirt-ci submodule and regenerate
  configure: never use PyPI for Meson
  python: use vendored tomli
  configure: switch to ensuregroup
  Revert "tests: Use separate virtual environment for avocado"
  tests/docker: add python3-tomli dependency to containers

 .gitlab-ci.d/buildtest.yml                    |   6 +-
 .gitlab-ci.d/cirrus/freebsd-13.vars           |   2 +-
 .gitlab-ci.d/cirrus/macos-12.vars             |   2 +-
 configure                                     |  22 +-
 docs/devel/acpi-bits.rst                      |   6 +-
 docs/devel/testing.rst                        |  14 +-
 python/scripts/mkvenv.py                      | 201 ++++++++++++++++--
 python/scripts/vendor.py                      |   5 +-
 python/setup.cfg                              |   9 +
 python/wheels/tomli-2.0.1-py3-none-any.whl    | Bin 0 -> 12757 bytes
 pythondeps.toml                               |  32 +++
 .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
 scripts/device-crash-test                     |   2 +-
 tests/Makefile.include                        |  19 +-
 tests/docker/dockerfiles/centos8.docker       |   3 +-
 .../dockerfiles/debian-all-test-cross.docker  |   7 +-
 .../dockerfiles/debian-amd64-cross.docker     |   6 +-
 tests/docker/dockerfiles/debian-amd64.docker  |   4 +
 .../dockerfiles/debian-arm64-cross.docker     |   6 +-
 .../dockerfiles/debian-armel-cross.docker     |   6 +-
 .../dockerfiles/debian-armhf-cross.docker     |   6 +-
 .../dockerfiles/debian-hexagon-cross.docker   |   6 +-
 .../dockerfiles/debian-mips64el-cross.docker  |   6 +-
 .../dockerfiles/debian-mipsel-cross.docker    |   6 +-
 .../dockerfiles/debian-ppc64el-cross.docker   |   6 +-
 .../dockerfiles/debian-riscv64-cross.docker   |   2 +-
 .../dockerfiles/debian-s390x-cross.docker     |   6 +-
 .../dockerfiles/debian-tricore-cross.docker   |   2 +
 .../dockerfiles/fedora-i386-cross.docker      |   1 +
 .../dockerfiles/fedora-win32-cross.docker     |   2 +-
 .../dockerfiles/fedora-win64-cross.docker     |   2 +-
 tests/docker/dockerfiles/opensuse-leap.docker |  23 +-
 tests/docker/dockerfiles/ubuntu2004.docker    |   4 +-
 tests/docker/dockerfiles/ubuntu2204.docker    |   1 +
 tests/lcitool/libvirt-ci                      |   2 +-
 tests/lcitool/mappings.yml                    |  27 ++-
 tests/lcitool/projects/qemu.yml               |   3 +-
 tests/lcitool/targets/opensuse-leap-15.yml    |   4 +-
 tests/requirements.txt                        |   6 -
 tests/vm/Makefile.include                     |   2 +-
 tests/vm/generated/freebsd.json               |   1 +
 41 files changed, 371 insertions(+), 103 deletions(-)
 create mode 100644 python/wheels/tomli-2.0.1-py3-none-any.whl
 create mode 100644 pythondeps.toml
 delete mode 100644 tests/requirements.txt

-- 
2.41.0
Re: [PATCH 0/9] Use known good releases when installing in pyvenv
Posted by John Snow 9 months, 2 weeks ago
On Wed, Aug 9, 2023 at 5:29 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> This series introduce a new installation command for mkvenv.py that
> retrieves the packages to be installed from a TOML file. This allows
> being more flexible in using the system version of a package, while at
> the same time using a known-good version when installing the package.
> This is important for packages that sometimes have backwards-incompatible
> changes or that depend on specific versions of their dependencies.
>
> For example, in the case of sphinx we can always use the last version
> that supports older versions of Python and especially docutils (Ubuntu
> 20.04 for example would not support sphinx 6.0), and in the case of
> Avocado we can avoid installing a package that conflicts with any
> plugins already existing in the system package path.  This in turn
> enables using the same virtual environment for both the build and
> qemu-iotests.  John has patches for this.
>
> For the configuration file, TOML was chosen because it was more human
> readable and easier to edit than JSON.  A parser is available in the
> Python 3.11 standard library; for older versions, the API-compatible
> replacement tomli is very small (12k).  I am introducing it as a vendored
> .whl file because it is not installed by default in most distros (unlike
> pip and setuptools which were introduced in 8.0) and because Debian
> 11 only has it in bullseye-backports.  However, if preferred the patch
> "python: use vendored tomli" can be dropped.  In that case, tomli will
> have to be installed from either PyPI or bullseye-backports.
>
> While tomli is bundled with pip, this is only true of recent versions
> of pip.  Of all the supported OSes pretty much only FreeBSD has a recent
> enough version of pip while staying on Python <3.11.  So we cannot use
> the same trick that is in place for distlib.
>
> In order to pick the tomli mapping, lcitool is updated to a recent
> version.  As a side effect this updates from LEAP 15.4 to 15.5.
>
> Paolo

I'm still nose to the grindstone trying to finish up the asyncio
machine demo, but I made good progress today and I'll try to get a
good look at this ASAP.
(But, I suppose we're both about to pop out for PTO soon, so...)

>
>
> Paolo Bonzini (9):
>   python: mkvenv: tweak the matching of --diagnose to depspecs
>   python: mkvenv: introduce TOML-like representation of dependencies
>   python: mkvenv: add ensuregroup command
>   lcitool: bump libvirt-ci submodule and regenerate
>   configure: never use PyPI for Meson
>   python: use vendored tomli
>   configure: switch to ensuregroup
>   Revert "tests: Use separate virtual environment for avocado"
>   tests/docker: add python3-tomli dependency to containers
>
>  .gitlab-ci.d/buildtest.yml                    |   6 +-
>  .gitlab-ci.d/cirrus/freebsd-13.vars           |   2 +-
>  .gitlab-ci.d/cirrus/macos-12.vars             |   2 +-
>  configure                                     |  22 +-
>  docs/devel/acpi-bits.rst                      |   6 +-
>  docs/devel/testing.rst                        |  14 +-
>  python/scripts/mkvenv.py                      | 201 ++++++++++++++++--
>  python/scripts/vendor.py                      |   5 +-
>  python/setup.cfg                              |   9 +
>  python/wheels/tomli-2.0.1-py3-none-any.whl    | Bin 0 -> 12757 bytes
>  pythondeps.toml                               |  32 +++
>  .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
>  scripts/device-crash-test                     |   2 +-
>  tests/Makefile.include                        |  19 +-
>  tests/docker/dockerfiles/centos8.docker       |   3 +-
>  .../dockerfiles/debian-all-test-cross.docker  |   7 +-
>  .../dockerfiles/debian-amd64-cross.docker     |   6 +-
>  tests/docker/dockerfiles/debian-amd64.docker  |   4 +
>  .../dockerfiles/debian-arm64-cross.docker     |   6 +-
>  .../dockerfiles/debian-armel-cross.docker     |   6 +-
>  .../dockerfiles/debian-armhf-cross.docker     |   6 +-
>  .../dockerfiles/debian-hexagon-cross.docker   |   6 +-
>  .../dockerfiles/debian-mips64el-cross.docker  |   6 +-
>  .../dockerfiles/debian-mipsel-cross.docker    |   6 +-
>  .../dockerfiles/debian-ppc64el-cross.docker   |   6 +-
>  .../dockerfiles/debian-riscv64-cross.docker   |   2 +-
>  .../dockerfiles/debian-s390x-cross.docker     |   6 +-
>  .../dockerfiles/debian-tricore-cross.docker   |   2 +
>  .../dockerfiles/fedora-i386-cross.docker      |   1 +
>  .../dockerfiles/fedora-win32-cross.docker     |   2 +-
>  .../dockerfiles/fedora-win64-cross.docker     |   2 +-
>  tests/docker/dockerfiles/opensuse-leap.docker |  23 +-
>  tests/docker/dockerfiles/ubuntu2004.docker    |   4 +-
>  tests/docker/dockerfiles/ubuntu2204.docker    |   1 +
>  tests/lcitool/libvirt-ci                      |   2 +-
>  tests/lcitool/mappings.yml                    |  27 ++-
>  tests/lcitool/projects/qemu.yml               |   3 +-
>  tests/lcitool/targets/opensuse-leap-15.yml    |   4 +-
>  tests/requirements.txt                        |   6 -
>  tests/vm/Makefile.include                     |   2 +-
>  tests/vm/generated/freebsd.json               |   1 +
>  41 files changed, 371 insertions(+), 103 deletions(-)
>  create mode 100644 python/wheels/tomli-2.0.1-py3-none-any.whl
>  create mode 100644 pythondeps.toml
>  delete mode 100644 tests/requirements.txt
>
> --
> 2.41.0
>
Re: [PATCH 0/9] Use known good releases when installing in pyvenv
Posted by Philippe Mathieu-Daudé 9 months, 2 weeks ago
Hi Paolo,

On 9/8/23 11:29, Paolo Bonzini wrote:
> This series introduce a new installation command for mkvenv.py that
> retrieves the packages to be installed from a TOML file. This allows
> being more flexible in using the system version of a package, while at
> the same time using a known-good version when installing the package.
> This is important for packages that sometimes have backwards-incompatible
> changes or that depend on specific versions of their dependencies.
>      
> For example, in the case of sphinx we can always use the last version
> that supports older versions of Python and especially docutils (Ubuntu
> 20.04 for example would not support sphinx 6.0), and in the case of
> Avocado we can avoid installing a package that conflicts with any
> plugins already existing in the system package path.  This in turn
> enables using the same virtual environment for both the build and
> qemu-iotests.  John has patches for this.
> 
> For the configuration file, TOML was chosen because it was more human
> readable and easier to edit than JSON.  A parser is available in the
> Python 3.11 standard library; for older versions, the API-compatible
> replacement tomli is very small (12k).  I am introducing it as a vendored
> .whl file because it is not installed by default in most distros (unlike
> pip and setuptools which were introduced in 8.0) and because Debian
> 11 only has it in bullseye-backports.  However, if preferred the patch
> "python: use vendored tomli" can be dropped.  In that case, tomli will
> have to be installed from either PyPI or bullseye-backports.
> 
> While tomli is bundled with pip, this is only true of recent versions
> of pip.  Of all the supported OSes pretty much only FreeBSD has a recent
> enough version of pip while staying on Python <3.11.  So we cannot use
> the same trick that is in place for distlib.
> 
> In order to pick the tomli mapping, lcitool is updated to a recent
> version.  As a side effect this updates from LEAP 15.4 to 15.5.

Does this fix the second point of commit eaf245becd ("Revert
"tests/requirements.txt: bump up avocado-framework version to 101.0"")

  > Second, the recent changes to Python handling mean that there is
  > a single virtual environment for all the build, instead of a separate
  > one for testing.

?

If so, what's left to be able to run QEMU (Avocado) test suite on macOS
is the first point from the same commit:

  > Avocado is not logging of command lines or terminal output, and
  > not collecting Python logs outside the avocado namespace.

This seems to be fixed in Avocado 102.0:

https://avocado-framework.readthedocs.io/en/102.0/releases/102_0.html

  > The root logger handler was restored. This enables all loggers
  > out of avocado.* namespace by default.

Assuming this is enough for macOS, what would be the next change,
moving to 102.0 or integrate 102.0 in the current set? I.e.:

   avocado-framework = { accepted = "(>=88.1, <93.0, 102.0)", ... }

? But I expect the "<93.0" filter to clash...

Thanks,

Phil.
Re: [PATCH 0/9] Use known good releases when installing in pyvenv
Posted by Paolo Bonzini 9 months, 2 weeks ago
On Thu, Aug 10, 2023 at 1:52 PM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
> Does this fix the second point of commit eaf245becd ("Revert
> "tests/requirements.txt: bump up avocado-framework version to 101.0"")
>
>   > Second, the recent changes to Python handling mean that there is
>   > a single virtual environment for all the build, instead of a separate
>   > one for testing.

This was reverted and Avocado has a separate venv than pyvenv/ (which
is the one created by configure). This series makes it possible to
have again a single virtual environment for all the build.

>   > Avocado is not logging of command lines or terminal output, and
>   > not collecting Python logs outside the avocado namespace.
>
> This seems to be fixed in Avocado 102.0:

Correct.

> Assuming this is enough for macOS, what would be the next change,
> moving to 102.0 or integrate 102.0 in the current set? I.e.:
>
>    avocado-framework = { accepted = "(>=88.1, <93.0, 102.0)", ... }

The plan is to:

1) wait for the next LTS release of Avocado (103.0)

2) wait for it to be packaged in Fedora

3) change "accepted" to be "~= 103.0"

Users of Fedora will have to switch from the 92.0 module to the 103.0
module if they want to use the system avocado.

Paolo