Currently we have "sphinx" and "sphinx_rtd_theme" in a single
group in pythondeps.toml. This means that mkvenv will try to
install them both at once, which doesn't work if sphinx_rtd_theme
depends on a version of Sphinx which is newer than the system
one, even if the "sphinx" we're installing at the same time
would satisfy that dependency. For instance this:

 sphinx = { accepted = ">=8.0.0rc1", installed = "8.0.0rc1", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=2.1.0rc1", installed = "2.1.0rc1" }

fails like this:

mkvenv: Creating non-isolated virtual environment at 'pyvenv'
mkvenv: checking for meson>=1.1.0
mkvenv: installing meson==1.2.3
mkvenv: checking for sphinx>=8.0.0rc1
mkvenv: checking for sphinx_rtd_theme>=2.1.0rc1
mkvenv: installing sphinx==8.0.0rc1, sphinx_rtd_theme==2.1.0rc1
ERROR: Cannot install sphinx-rtd-theme==2.1.0rc1 and sphinx==8.0.0rc1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

*** Ouch! ***

Could not provide build dependency 'sphinx==8.0.0rc1':
 • Python package 'sphinx' version '4.3.2' was found, but isn't suitable.
 • A suitable version could not be obtained from PyPI.

(where 4.3.2 is the system version of sphinx on my system).

Avoid this problem by splitting sphinx_rtd_theme into its own
group in pythondeps, so we can install the new sphinx into the
venv first and the sphinx_rtd_theme afterwards.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
I'm just bumbling around in the dark here, hence the RFC tag: maybe
there's a better way to fix this? I ran into this when I was trying
to test the fix for https://gitlab.com/qemu-project/qemu/-/issues/2458
and wanted a newer Sphinx to test with. Unless you also install
the newer RTD theme the new Sphinx will emit some deprecation warnings
about things the theme is doing.
---
 configure       | 7 +++++--
 pythondeps.toml | 3 +++
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index 019fcbd0ef7..7f9cfbffa2c 100755
--- a/configure
+++ b/configure
@@ -973,9 +973,12 @@ if test "$download" = "enabled" ; then
 fi
 
 if test "$docs" != "disabled" ; then
-    if ! $mkvenv ensuregroup \
+    if ! ( $mkvenv ensuregroup \
          $(test "$docs" = "enabled" && echo "$mkvenv_online_flag") \
-         ${source_path}/pythondeps.toml docs;
+         ${source_path}/pythondeps.toml docs && \
+         $mkvenv ensuregroup \
+         $(test "$docs" = "enabled" && echo "$mkvenv_online_flag") \
+         ${source_path}/pythondeps.toml docs-theme );
     then
         if test "$docs" = "enabled" ; then
             exit 1
diff --git a/pythondeps.toml b/pythondeps.toml
index f6e590fdd86..fdf5579c43e 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -24,6 +24,9 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
 sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
+
+[docs-theme]
+# Please keep the installed versions in sync with docs/requirements.txt
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
 [avocado]
-- 
2.34.1

On Mon, Jul 29, 2024 at 2:26 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> Currently we have "sphinx" and "sphinx_rtd_theme" in a single
> group in pythondeps.toml. This means that mkvenv will try to
> install them both at once, which doesn't work if sphinx_rtd_theme
> depends on a version of Sphinx which is newer than the system
> one, even if the "sphinx" we're installing at the same time
> would satisfy that dependency. For instance this:
>
>  sphinx = { accepted = ">=8.0.0rc1", installed = "8.0.0rc1", canary = "sphinx-build" }
>  sphinx_rtd_theme = { accepted = ">=2.1.0rc1", installed = "2.1.0rc1" }
>
> fails like this:
>
> mkvenv: Creating non-isolated virtual environment at 'pyvenv'
> mkvenv: checking for meson>=1.1.0
> mkvenv: installing meson==1.2.3
> mkvenv: checking for sphinx>=8.0.0rc1
> mkvenv: checking for sphinx_rtd_theme>=2.1.0rc1
> mkvenv: installing sphinx==8.0.0rc1, sphinx_rtd_theme==2.1.0rc1
> ERROR: Cannot install sphinx-rtd-theme==2.1.0rc1 and sphinx==8.0.0rc1 because these package versions have conflicting dependencies.
> ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
>
> *** Ouch! ***
>
> Could not provide build dependency 'sphinx==8.0.0rc1':
>  • Python package 'sphinx' version '4.3.2' was found, but isn't suitable.
>  • A suitable version could not be obtained from PyPI.
>
> (where 4.3.2 is the system version of sphinx on my system).

I think the error is a bit confusing, but the versions seem to be
genuinely incompatibile. sphinx_rtd_theme is still incompatible with
sphinx 8.0.0, and the error is correct:

=========
$ python -m venv testvenv
$ cd testvenv
$ pip install 'sphinx==8.0.0rc1' 'sphinx_rtd_theme==2.1.0rc1'
Defaulting to user installation because normal site-packages is not writeable
Collecting sphinx==8.0.0rc1
  Downloading sphinx-8.0.0rc1-py3-none-any.whl.metadata (6.2 kB)
Collecting sphinx_rtd_theme==2.1.0rc1
  Downloading sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting sphinxcontrib-applehelp (from sphinx==8.0.0rc1)
  Downloading sphinxcontrib_applehelp-2.0.0-py3-none-any.whl.metadata (2.3 kB)
Requirement already satisfied: sphinxcontrib-devhelp in
/usr/lib/python3.12/site-packages (from sphinx==8.0.0rc1) (1.0.5)
[more "requirement already satisfied" lines...]
Requirement already satisfied: packaging>=23.0 in
/usr/lib/python3.12/site-packages (from sphinx==8.0.0rc1) (23.2)
INFO: pip is looking at multiple versions of sphinx-rtd-theme to
determine which version is compatible with other requirements. This
could take a while.
ERROR: Cannot install sphinx-rtd-theme==2.1.0rc1 and sphinx==8.0.0rc1
because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sphinx==8.0.0rc1
    sphinx-rtd-theme 2.1.0rc1 depends on sphinx<8 and >=5
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit
https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
=========

The reason why your patch works is weird, and it's apparent when you
run the "pip install" commands one by one:

=========
$ pip install 'sphinx_rtd_theme==2.1.0rc1'
Defaulting to user installation because normal site-packages is not writeable
Collecting sphinx_rtd_theme==2.1.0rc1
  Using cached sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting sphinx<8,>=5 (from sphinx_rtd_theme==2.1.0rc1)
  Downloading sphinx-7.4.7-py3-none-any.whl.metadata (6.1 kB)
Requirement already satisfied: docutils<0.21 in
/usr/lib/python3.12/site-packages (from sphinx_rtd_theme==2.1.0rc1)
(0.20.1)
[more "requirement already satisfied" lines...]
Downloading sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl (7.7 MB)
Downloading sphinx-7.4.7-py3-none-any.whl (3.4 MB)
Installing collected packages: sphinx, sphinx_rtd_theme
  Attempting uninstall: sphinx
    Found existing installation: Sphinx 8.0.0rc1
    Uninstalling Sphinx-8.0.0rc1:
      Successfully uninstalled Sphinx-8.0.0rc1
Successfully installed sphinx-7.4.7 sphinx_rtd_theme-2.1.0rc1
=========

It doesn't install 8.0.0rc1 at all... :)

Paolo

On Tue, 30 Jul 2024 at 15:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On Mon, Jul 29, 2024 at 2:26 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > Currently we have "sphinx" and "sphinx_rtd_theme" in a single
> > group in pythondeps.toml. This means that mkvenv will try to
> > install them both at once, which doesn't work if sphinx_rtd_theme
> > depends on a version of Sphinx which is newer than the system
> > one, even if the "sphinx" we're installing at the same time
> > would satisfy that dependency. For instance this:
> >
> >  sphinx = { accepted = ">=8.0.0rc1", installed = "8.0.0rc1", canary = "sphinx-build" }
> >  sphinx_rtd_theme = { accepted = ">=2.1.0rc1", installed = "2.1.0rc1" }
> >
> > fails like this:
> >
> > mkvenv: Creating non-isolated virtual environment at 'pyvenv'
> > mkvenv: checking for meson>=1.1.0
> > mkvenv: installing meson==1.2.3
> > mkvenv: checking for sphinx>=8.0.0rc1
> > mkvenv: checking for sphinx_rtd_theme>=2.1.0rc1
> > mkvenv: installing sphinx==8.0.0rc1, sphinx_rtd_theme==2.1.0rc1
> > ERROR: Cannot install sphinx-rtd-theme==2.1.0rc1 and sphinx==8.0.0rc1 because these package versions have conflicting dependencies.
> > ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
> >
> > *** Ouch! ***
> >
> > Could not provide build dependency 'sphinx==8.0.0rc1':
> >  • Python package 'sphinx' version '4.3.2' was found, but isn't suitable.
> >  • A suitable version could not be obtained from PyPI.
> >
> > (where 4.3.2 is the system version of sphinx on my system).
>
> I think the error is a bit confusing, but the versions seem to be
> genuinely incompatibile. sphinx_rtd_theme is still incompatible with
> sphinx 8.0.0, and the error is correct:
>
> =========
> $ python -m venv testvenv
> $ cd testvenv
> $ pip install 'sphinx==8.0.0rc1' 'sphinx_rtd_theme==2.1.0rc1'
> Defaulting to user installation because normal site-packages is not writeable
> Collecting sphinx==8.0.0rc1
>   Downloading sphinx-8.0.0rc1-py3-none-any.whl.metadata (6.2 kB)
> Collecting sphinx_rtd_theme==2.1.0rc1
>   Downloading sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl.metadata (4.4 kB)
> Collecting sphinxcontrib-applehelp (from sphinx==8.0.0rc1)
>   Downloading sphinxcontrib_applehelp-2.0.0-py3-none-any.whl.metadata (2.3 kB)
> Requirement already satisfied: sphinxcontrib-devhelp in
> /usr/lib/python3.12/site-packages (from sphinx==8.0.0rc1) (1.0.5)
> [more "requirement already satisfied" lines...]
> Requirement already satisfied: packaging>=23.0 in
> /usr/lib/python3.12/site-packages (from sphinx==8.0.0rc1) (23.2)
> INFO: pip is looking at multiple versions of sphinx-rtd-theme to
> determine which version is compatible with other requirements. This
> could take a while.
> ERROR: Cannot install sphinx-rtd-theme==2.1.0rc1 and sphinx==8.0.0rc1
> because these package versions have conflicting dependencies.
>
> The conflict is caused by:
>     The user requested sphinx==8.0.0rc1
>     sphinx-rtd-theme 2.1.0rc1 depends on sphinx<8 and >=5
> To fix this you could try to:
> 1. loosen the range of package versions you've specified
> 2. remove package versions to allow pip attempt to solve the dependency conflict
>
> ERROR: ResolutionImpossible: for help visit
> https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
> =========
>
> The reason why your patch works is weird, and it's apparent when you
> run the "pip install" commands one by one:
>
> =========
> $ pip install 'sphinx_rtd_theme==2.1.0rc1'
> Defaulting to user installation because normal site-packages is not writeable
> Collecting sphinx_rtd_theme==2.1.0rc1
>   Using cached sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl.metadata (4.4 kB)
> Collecting sphinx<8,>=5 (from sphinx_rtd_theme==2.1.0rc1)
>   Downloading sphinx-7.4.7-py3-none-any.whl.metadata (6.1 kB)
> Requirement already satisfied: docutils<0.21 in
> /usr/lib/python3.12/site-packages (from sphinx_rtd_theme==2.1.0rc1)
> (0.20.1)
> [more "requirement already satisfied" lines...]
> Downloading sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl (7.7 MB)
> Downloading sphinx-7.4.7-py3-none-any.whl (3.4 MB)
> Installing collected packages: sphinx, sphinx_rtd_theme
>   Attempting uninstall: sphinx
>     Found existing installation: Sphinx 8.0.0rc1
>     Uninstalling Sphinx-8.0.0rc1:
>       Successfully uninstalled Sphinx-8.0.0rc1
> Successfully installed sphinx-7.4.7 sphinx_rtd_theme-2.1.0rc1
> =========
>
> It doesn't install 8.0.0rc1 at all... :)

Aha. Is it possible to get mkvenv to produce these full
error messages rather than the truncated version?

thanks
-- PMM

> > The reason why your patch works is weird, and it's apparent when you
> > run the "pip install" commands one by one:
> >
> > =========
> > $ pip install 'sphinx_rtd_theme==2.1.0rc1'
> > Defaulting to user installation because normal site-packages is not writeable
> > Collecting sphinx_rtd_theme==2.1.0rc1
> >   Using cached sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl.metadata (4.4 kB)
> > Collecting sphinx<8,>=5 (from sphinx_rtd_theme==2.1.0rc1)
> >   Downloading sphinx-7.4.7-py3-none-any.whl.metadata (6.1 kB)
> > Requirement already satisfied: docutils<0.21 in
> > /usr/lib/python3.12/site-packages (from sphinx_rtd_theme==2.1.0rc1)
> > (0.20.1)
> > [more "requirement already satisfied" lines...]
> > Downloading sphinx_rtd_theme-2.1.0rc1-py2.py3-none-any.whl (7.7 MB)
> > Downloading sphinx-7.4.7-py3-none-any.whl (3.4 MB)
> > Installing collected packages: sphinx, sphinx_rtd_theme
> >   Attempting uninstall: sphinx
> >     Found existing installation: Sphinx 8.0.0rc1
> >     Uninstalling Sphinx-8.0.0rc1:
> >       Successfully uninstalled Sphinx-8.0.0rc1
> > Successfully installed sphinx-7.4.7 sphinx_rtd_theme-2.1.0rc1
> > =========
> >
> > It doesn't install 8.0.0rc1 at all... :)
>
> Aha. Is it possible to get mkvenv to produce these full
> error messages rather than the truncated version?

Heh, the idea of pythondeps.toml was that they wouldn't be necessary -
hence the messages that are slightly more tailored to the
QEMU-specific issue. It's certainly a good idea to at least place them
in a venv-pip.log file or something like that.

Paolo