Howdy, this series increases our minimum python version to 3.7.

CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
    (All green!)
GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37

Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
qemu.git within the week whether or not we take this series. I'd
appreciate an "ACK" on those specifically. They're just riding along
here because they make this series a bit nicer.

Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.

The motivation for this series is that Python 3.6 was EOL at the end of
2021; upstream tools are beginning to drop support for it, including
setuptools, pylint, mypy, etc. As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions.

The qemu.qmp library and the avocado testing framework both have
motivations for dropping 3.6 support, but are committed to not doing so
until QEMU drops support.

So, I'd like to talk about doing it.

V2:
- Added R-Bs to patch 1
- Updated commit message for patch 7 with explicit version info
- Added DO-NOT-MERGE to patch 5's title
- Tested tests/vm/freebsd, netbsd, and openbsd in addition to full CI

RFC:
 - Patch 5 is just a proof-of-concept; we need to update lcitool instead.
 - Cleber, I need to update your ansible scripts. How do I test them?

Thanks!
--js

John Snow (7):
  python: support pylint 2.16
  python: drop pipenv
  configure: Look for auxiliary Python installations
  configure: Add nice hint to Python failure message
  DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE
  CI: Stop building docs on centos8
  Python: Drop support for Python 3.6

 docs/conf.py                                  |   4 +-
 python/README.rst                             |   3 -
 configure                                     |  40 +-
 .gitlab-ci.d/buildtest.yml                    |   2 +-
 .gitlab-ci.d/static_checks.yml                |   4 +-
 python/.gitignore                             |   4 +-
 python/Makefile                               |  57 ++-
 python/Pipfile                                |  13 -
 python/Pipfile.lock                           | 347 ------------------
 python/qemu/qmp/protocol.py                   |   2 +-
 python/qemu/qmp/qmp_client.py                 |   2 +-
 python/qemu/utils/qemu_ga_client.py           |   6 +-
 python/setup.cfg                              |  11 +-
 python/tests/minreqs.txt                      |  45 +++
 scripts/qapi/mypy.ini                         |   2 +-
 tests/docker/dockerfiles/centos8.docker       |   1 +
 tests/docker/dockerfiles/opensuse-leap.docker |   1 +
 tests/docker/dockerfiles/python.docker        |   1 -
 tests/qemu-iotests/iotests.py                 |   4 +-
 .../tests/migrate-bitmaps-postcopy-test       |   2 +-
 20 files changed, 135 insertions(+), 416 deletions(-)
 delete mode 100644 python/Pipfile
 delete mode 100644 python/Pipfile.lock
 create mode 100644 python/tests/minreqs.txt

-- 
2.39.0

The discussion under PATCH 6 makes me think there's a bit of confusion
about the actual impact of dropping support for Python 3.6.  Possibly
because it's spelled out in the commit message of PATCH 7.  Let me
summarize it in one sentence:

    *** All supported host systems continue to work ***

Evidence: CI remains green.

On some supported host systems, different packages need to be installed.
On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
instead of 3.6.8.  Let me stress again: same repository, different
package.  No downsides I can see.

The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
version of Sphinx that works with Python 3.7 or newer.  This series
proposes to simply stop building the docs there, unless the user
provides a suitable version of Sphinx (which is easy enough with pip).
That's PATCH 7.

Paolo thinks we could also stay on 3.6 just for Sphinx, with a bit of
care.  That's constructive patch review.

All the angst about CentOS falling off the end of our "supported build
platforms" list is not actually warranted by this series :)

Some of the discussion is valuable regardless.  For instance, the points
Daniel made about traditional distro dependencies (still used with C
tool chains), and distro-agnostic dependencies baked into modern tool
chains.  This series is not (and does not try to be) the final answer to
the question how to handle Python dependencies in QEMU.

On 15/02/2023 20.05, Markus Armbruster wrote:
> The discussion under PATCH 6 makes me think there's a bit of confusion
> about the actual impact of dropping support for Python 3.6.  Possibly
> because it's spelled out in the commit message of PATCH 7.  Let me
> summarize it in one sentence:
> 
>      *** All supported host systems continue to work ***
> 
> Evidence: CI remains green.

The CI remains green since one of the patches disabled the building of the 
docs on CentOS 8. That's not how I'd describe "continue to work", at least 
not in the same extend as before.

> On some supported host systems, different packages need to be installed.
> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> instead of 3.6.8.  Let me stress again: same repository, different
> package.  No downsides I can see.
> 
> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> version of Sphinx that works with Python 3.7 or newer.  This series
> proposes to simply stop building the docs there, unless the user
> provides a suitable version of Sphinx (which is easy enough with pip).

I think we've all understood that. The thing that you obviously did not 
understood: This breaks our support statement.
I'm pretty sure that you could also build the whole QEMU suite successfully 
on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a 
newer version of GCC and some of the required libraries first. But that's 
not how we understand our support statement.

Sure, you can argue that you can use "pip install" to get a newer version of 
Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is 
that really that much different from installing a newer version of GCC and 
libraries on an ancient distro that we do not officially support anymore? 
I'd say no. You also have to consider that not every build host has access 
to the internet, maybe some companies only have an internal mirror of the 
distro packages in their intranet (I remember some discussion about such a 
case in the past) - so while you were perfectly fine to build the whole of 
QEMU on a CentOS 8 there before this change, you could now not build parts 
of QEMU anymore there due to the missing possibility to run "pip install" 
without full internet connection.

And sure, you can argue that it's "just" the documentation. But IMHO that's 
still an essential part of the QEMU build, and it used to work before, so it 
feels wrong to cut that now out. And also, if we start with the 
documentation now, what's next? If for example scripts/shaderinclude.py 
stops working with Python 3.6, do we then simply say: "Oh, it's fine, you 
can still build all the other targets that work without this script, just 
not the ones anymore that need it"?

> All the angst about CentOS falling off the end of our "supported build
> platforms" list is not actually warranted by this series :)

Using the term "angst" for the concerns of your fellows here is quite 
cheeky. It's not about "angst", it's about a discussion that our support 
policy might need to be adjusted if we do this step. So instead of writing 
such sentences, I'd rather would like to see you posting a patch for 
docs/about/build-platforms.rst for constructive further discussion instead.

  Thanks,
   Thomas

On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 15/02/2023 20.05, Markus Armbruster wrote:
> > The discussion under PATCH 6 makes me think there's a bit of confusion
> > about the actual impact of dropping support for Python 3.6.  Possibly
> > because it's spelled out in the commit message of PATCH 7.  Let me
> > summarize it in one sentence:
> >
> >      *** All supported host systems continue to work ***
> >
> > Evidence: CI remains green.
>
> The CI remains green since one of the patches disabled the building of the
> docs on CentOS 8. That's not how I'd describe "continue to work", at least
> not in the same extend as before.
>
> > On some supported host systems, different packages need to be installed.
> > On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> > instead of 3.6.8.  Let me stress again: same repository, different
> > package.  No downsides I can see.
> >
> > The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> > version of Sphinx that works with Python 3.7 or newer.  This series
> > proposes to simply stop building the docs there, unless the user
> > provides a suitable version of Sphinx (which is easy enough with pip).
>
> I think we've all understood that. The thing that you obviously did not
> understood: This breaks our support statement.
> I'm pretty sure that you could also build the whole QEMU suite successfully
> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
> newer version of GCC and some of the required libraries first. But that's
> not how we understand our support statement.
>
> Sure, you can argue that you can use "pip install" to get a newer version of
> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
> that really that much different from installing a newer version of GCC and
> libraries on an ancient distro that we do not officially support anymore?
> I'd say no. You also have to consider that not every build host has access
> to the internet, maybe some companies only have an internal mirror of the
> distro packages in their intranet (I remember some discussion about such a
> case in the past) - so while you were perfectly fine to build the whole of
> QEMU on a CentOS 8 there before this change, you could now not build parts
> of QEMU anymore there due to the missing possibility to run "pip install"
> without full internet connection.

There are good points elsewhere in this thread and I am taking notes,
but this critique caught my eye as something I was not specifically
planning around, so I wanted to get an elaboration here if I may.

Do we have a support statement for this? I find this critique somewhat
surprising -- If we don't have internet, how did we get the other 20
to 30 dependencies needed to build QEMU? To what extent are we
*required* to preserve a build that works without internet access?

you generally need internet to run "dnf install", as you would to "pip
install", so how does this distinction exclude one but not the other?

If you mean to say: "The build cannot rely on using internet-connected
pip to configure an environment just-in-time during a build because
internet may not be present" -- I completely agree, as this is a
necessity for e.g. RHEL packaging downstream. That requirement won't
be violated by me.

--js

On 17/02/2023 21.46, John Snow wrote:
> On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 15/02/2023 20.05, Markus Armbruster wrote:
>>> The discussion under PATCH 6 makes me think there's a bit of confusion
>>> about the actual impact of dropping support for Python 3.6.  Possibly
>>> because it's spelled out in the commit message of PATCH 7.  Let me
>>> summarize it in one sentence:
>>>
>>>       *** All supported host systems continue to work ***
>>>
>>> Evidence: CI remains green.
>>
>> The CI remains green since one of the patches disabled the building of the
>> docs on CentOS 8. That's not how I'd describe "continue to work", at least
>> not in the same extend as before.
>>
>>> On some supported host systems, different packages need to be installed.
>>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>>> instead of 3.6.8.  Let me stress again: same repository, different
>>> package.  No downsides I can see.
>>>
>>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>>> version of Sphinx that works with Python 3.7 or newer.  This series
>>> proposes to simply stop building the docs there, unless the user
>>> provides a suitable version of Sphinx (which is easy enough with pip).
>>
>> I think we've all understood that. The thing that you obviously did not
>> understood: This breaks our support statement.
>> I'm pretty sure that you could also build the whole QEMU suite successfully
>> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
>> newer version of GCC and some of the required libraries first. But that's
>> not how we understand our support statement.
>>
>> Sure, you can argue that you can use "pip install" to get a newer version of
>> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
>> that really that much different from installing a newer version of GCC and
>> libraries on an ancient distro that we do not officially support anymore?
>> I'd say no. You also have to consider that not every build host has access
>> to the internet, maybe some companies only have an internal mirror of the
>> distro packages in their intranet (I remember some discussion about such a
>> case in the past) - so while you were perfectly fine to build the whole of
>> QEMU on a CentOS 8 there before this change, you could now not build parts
>> of QEMU anymore there due to the missing possibility to run "pip install"
>> without full internet connection.
> 
> There are good points elsewhere in this thread and I am taking notes,
> but this critique caught my eye as something I was not specifically
> planning around, so I wanted to get an elaboration here if I may.
> 
> Do we have a support statement for this? I find this critique somewhat
> surprising -- If we don't have internet, how did we get the other 20
> to 30 dependencies needed to build QEMU? To what extent are we
> *required* to preserve a build that works without internet access?

It's not written in stone, but I saw it this way: If I have a complete 
mirror of a distro repository in my intrAnet, I can use that mirror to set 
up a QEMU build host system that has no access to the internet. Or maybe 
think of a DVD image(s) with all distro packages that you use to install a 
host without network access (and you copy the QEMU tarball there via USB 
stick). I think it's not that uncommon to have such scenarios out there.

For example, do you remember that SDL 1.2 discussion a some years ago? See:

  https://www.mail-archive.com/qemu-devel@nongnu.org/msg631628.html

It was not exactly the same situation, since those folks were even unable to 
install a SDL2-devel package on their pre-installed hosts, though it was 
theoretically available as an update in their distro, but I think it gives 
an impression of what people are using and expecting out there... (and no, 
I'm not happy with this, I'd also rather love if we could move faster in the 
QEMU project sometimes).

  Thomas

On Mon, Feb 20, 2023 at 1:16 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 17/02/2023 21.46, John Snow wrote:
> > On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
> >>
> >> On 15/02/2023 20.05, Markus Armbruster wrote:
> >>> The discussion under PATCH 6 makes me think there's a bit of confusion
> >>> about the actual impact of dropping support for Python 3.6.  Possibly
> >>> because it's spelled out in the commit message of PATCH 7.  Let me
> >>> summarize it in one sentence:
> >>>
> >>>       *** All supported host systems continue to work ***
> >>>
> >>> Evidence: CI remains green.
> >>
> >> The CI remains green since one of the patches disabled the building of the
> >> docs on CentOS 8. That's not how I'd describe "continue to work", at least
> >> not in the same extend as before.
> >>
> >>> On some supported host systems, different packages need to be installed.
> >>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> >>> instead of 3.6.8.  Let me stress again: same repository, different
> >>> package.  No downsides I can see.
> >>>
> >>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> >>> version of Sphinx that works with Python 3.7 or newer.  This series
> >>> proposes to simply stop building the docs there, unless the user
> >>> provides a suitable version of Sphinx (which is easy enough with pip).
> >>
> >> I think we've all understood that. The thing that you obviously did not
> >> understood: This breaks our support statement.
> >> I'm pretty sure that you could also build the whole QEMU suite successfully
> >> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
> >> newer version of GCC and some of the required libraries first. But that's
> >> not how we understand our support statement.
> >>
> >> Sure, you can argue that you can use "pip install" to get a newer version of
> >> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
> >> that really that much different from installing a newer version of GCC and
> >> libraries on an ancient distro that we do not officially support anymore?
> >> I'd say no. You also have to consider that not every build host has access
> >> to the internet, maybe some companies only have an internal mirror of the
> >> distro packages in their intranet (I remember some discussion about such a
> >> case in the past) - so while you were perfectly fine to build the whole of
> >> QEMU on a CentOS 8 there before this change, you could now not build parts
> >> of QEMU anymore there due to the missing possibility to run "pip install"
> >> without full internet connection.
> >
> > There are good points elsewhere in this thread and I am taking notes,
> > but this critique caught my eye as something I was not specifically
> > planning around, so I wanted to get an elaboration here if I may.
> >
> > Do we have a support statement for this? I find this critique somewhat
> > surprising -- If we don't have internet, how did we get the other 20
> > to 30 dependencies needed to build QEMU? To what extent are we
> > *required* to preserve a build that works without internet access?
>
> It's not written in stone, but I saw it this way: If I have a complete
> mirror of a distro repository in my intrAnet, I can use that mirror to set
> up a QEMU build host system that has no access to the internet. Or maybe
> think of a DVD image(s) with all distro packages that you use to install a
> host without network access (and you copy the QEMU tarball there via USB
> stick). I think it's not that uncommon to have such scenarios out there.
>
> For example, do you remember that SDL 1.2 discussion a some years ago? See:
>
>   https://www.mail-archive.com/qemu-devel@nongnu.org/msg631628.html
>
> It was not exactly the same situation, since those folks were even unable to
> install a SDL2-devel package on their pre-installed hosts, though it was
> theoretically available as an update in their distro, but I think it gives
> an impression of what people are using and expecting out there... (and no,
> I'm not happy with this, I'd also rather love if we could move faster in the
> QEMU project sometimes).
>
>   Thomas

Well, in this case I believe our support policy generally is written
to require a fully up-to-date version of the LTS distros, e.g. we
don't really test against "release day" 16.04, in the same way we
don't offer support for RHEL 8.0, just the latest point release. I
don't want to march things forward and break things for people for no
reason, but at a certain point, I have to ask: Why do people expect
software written three to four years after the release of their
operating system to not only run, but compile on that system -- with
no updates or internet? I think it's (unfortunately) reasonable to
expect that if you want to run a stable OS with no changes for years
that at a certain point, brand new releases may start requiring a few
hoops for you to jump through.

Or, in other words: If you can get code from 2019 onto a machine from
2016 to attempt to compile, you can also get the dependencies from the
future, too.

Still; with regards to the "offline building" thing specifically, it's
my intent to preserve the ability to build QEMU offline *provided* you
have the necessary dependencies in place already. For the Python case
under consideration, it would just be that you have your distro's
python38/python39 package installed. I consider this fundamentally no
different to other dependencies. For docs building it's a bit hairier;
you would indeed need a pip version installed prior to going offline.
The loss of docs doesn't fail the build, though; they aren't
*technically* required.

I think really all we need is the ability to know a priori what we
need to build QEMU before going offline without any last second
surprises during configure, make, or make check. Right? Or do we
really want to say "Any preparation that might be needed from outside
your system's repository *at all* is entirely prohibited"?

On 20/02/2023 20.56, John Snow wrote:
> On Mon, Feb 20, 2023 at 1:16 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 17/02/2023 21.46, John Snow wrote:
>>> On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>>>>
>>>> On 15/02/2023 20.05, Markus Armbruster wrote:
>>>>> The discussion under PATCH 6 makes me think there's a bit of confusion
>>>>> about the actual impact of dropping support for Python 3.6.  Possibly
>>>>> because it's spelled out in the commit message of PATCH 7.  Let me
>>>>> summarize it in one sentence:
>>>>>
>>>>>        *** All supported host systems continue to work ***
>>>>>
>>>>> Evidence: CI remains green.
>>>>
>>>> The CI remains green since one of the patches disabled the building of the
>>>> docs on CentOS 8. That's not how I'd describe "continue to work", at least
>>>> not in the same extend as before.
>>>>
>>>>> On some supported host systems, different packages need to be installed.
>>>>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>>>>> instead of 3.6.8.  Let me stress again: same repository, different
>>>>> package.  No downsides I can see.
>>>>>
>>>>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>>>>> version of Sphinx that works with Python 3.7 or newer.  This series
>>>>> proposes to simply stop building the docs there, unless the user
>>>>> provides a suitable version of Sphinx (which is easy enough with pip).
>>>>
>>>> I think we've all understood that. The thing that you obviously did not
>>>> understood: This breaks our support statement.
>>>> I'm pretty sure that you could also build the whole QEMU suite successfully
>>>> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
>>>> newer version of GCC and some of the required libraries first. But that's
>>>> not how we understand our support statement.
>>>>
>>>> Sure, you can argue that you can use "pip install" to get a newer version of
>>>> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
>>>> that really that much different from installing a newer version of GCC and
>>>> libraries on an ancient distro that we do not officially support anymore?
>>>> I'd say no. You also have to consider that not every build host has access
>>>> to the internet, maybe some companies only have an internal mirror of the
>>>> distro packages in their intranet (I remember some discussion about such a
>>>> case in the past) - so while you were perfectly fine to build the whole of
>>>> QEMU on a CentOS 8 there before this change, you could now not build parts
>>>> of QEMU anymore there due to the missing possibility to run "pip install"
>>>> without full internet connection.
>>>
>>> There are good points elsewhere in this thread and I am taking notes,
>>> but this critique caught my eye as something I was not specifically
>>> planning around, so I wanted to get an elaboration here if I may.
>>>
>>> Do we have a support statement for this? I find this critique somewhat
>>> surprising -- If we don't have internet, how did we get the other 20
>>> to 30 dependencies needed to build QEMU? To what extent are we
>>> *required* to preserve a build that works without internet access?
>>
>> It's not written in stone, but I saw it this way: If I have a complete
>> mirror of a distro repository in my intrAnet, I can use that mirror to set
>> up a QEMU build host system that has no access to the internet. Or maybe
>> think of a DVD image(s) with all distro packages that you use to install a
>> host without network access (and you copy the QEMU tarball there via USB
>> stick). I think it's not that uncommon to have such scenarios out there.
>>
>> For example, do you remember that SDL 1.2 discussion a some years ago? See:
>>
>>    https://www.mail-archive.com/qemu-devel@nongnu.org/msg631628.html
>>
>> It was not exactly the same situation, since those folks were even unable to
>> install a SDL2-devel package on their pre-installed hosts, though it was
>> theoretically available as an update in their distro, but I think it gives
>> an impression of what people are using and expecting out there... (and no,
>> I'm not happy with this, I'd also rather love if we could move faster in the
>> QEMU project sometimes).
>>
>>    Thomas
> 
> Well, in this case I believe our support policy generally is written
> to require a fully up-to-date version of the LTS distros, e.g. we
> don't really test against "release day" 16.04, in the same way we
> don't offer support for RHEL 8.0, just the latest point release.

Yes, sure, that's what I meant with "not exactly the same situation" ... it 
was just an example of people trying to compile QEMU offline.

> I think really all we need is the ability to know a priori what we
> need to build QEMU before going offline without any last second
> surprises during configure, make, or make check. Right?

I think it should be OK with the patch that Paolo suggested for the support 
policy and maybe a note somewhere that you have to make sure to install a 
newer Sphinx with pip in case you still want to build the docs on older 
enterprise distros...

  Thomas

Thomas Huth <thuth@redhat.com> writes:

> On 15/02/2023 20.05, Markus Armbruster wrote:
>> The discussion under PATCH 6 makes me think there's a bit of confusion
>> about the actual impact of dropping support for Python 3.6.  Possibly
>> because it's spelled out in the commit message of PATCH 7.  Let me
>> summarize it in one sentence:
>> 
>>      *** All supported host systems continue to work ***
>> 
>> Evidence: CI remains green.
>
> The CI remains green since one of the patches disabled the building of the 
> docs on CentOS 8. That's not how I'd describe "continue to work", at least 
> not in the same extend as before.

Thus the exception ...

>> On some supported host systems, different packages need to be installed.
>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>> instead of 3.6.8.  Let me stress again: same repository, different
>> package.  No downsides I can see.

... right here:

>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>> version of Sphinx that works with Python 3.7 or newer.  This series
>> proposes to simply stop building the docs there, unless the user
>> provides a suitable version of Sphinx (which is easy enough with pip).
>
> I think we've all understood that. The thing that you obviously did not 
> understood: This breaks our support statement.
> I'm pretty sure that you could also build the whole QEMU suite successfully 
> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a 
> newer version of GCC and some of the required libraries first. But that's 
> not how we understand our support statement.
>
> Sure, you can argue that you can use "pip install" to get a newer version of 
> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is 
> that really that much different from installing a newer version of GCC and 
> libraries on an ancient distro that we do not officially support anymore? 
> I'd say no. You also have to consider that not every build host has access 
> to the internet, maybe some companies only have an internal mirror of the 
> distro packages in their intranet (I remember some discussion about such a 
> case in the past) - so while you were perfectly fine to build the whole of 
> QEMU on a CentOS 8 there before this change, you could now not build parts 
> of QEMU anymore there due to the missing possibility to run "pip install" 
> without full internet connection.
>
> And sure, you can argue that it's "just" the documentation. But IMHO that's 
> still an essential part of the QEMU build, and it used to work before, so it 
> feels wrong to cut that now out. And also, if we start with the 
> documentation now, what's next? If for example scripts/shaderinclude.py 
> stops working with Python 3.6, do we then simply say: "Oh, it's fine, you 
> can still build all the other targets that work without this script, just 
> not the ones anymore that need it"?

My view on all this is a bit more pragmatic.

For a human developer, the difference between "dnf install
python-sphinx" and "pip install sphinx" is, in my opinion, close to
negligible.  Really no comparison to "git-clone GCC and bootstap it".
You seem to disagree with that.

For automated builds in general, and distro packaging in particular, the
difference is real, and could even be a show stopper.  But who's
packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
builds are our own CI, where the difference is real, but hardly a show
stopper.  John's patch is the stupidest solution that could possibly
work for us: disable doc building on CentOS 8.  Alternative solutions
have been proposed, and that's fair.  Again, you seem to think this
issue is a lot more serious.

So maybe this breaks our support statement for a sufficiently rigid
interpretation of our support statement.  Not the way interpreted it,
but if it's the way it is to be interpreted, I stand corrected.

But then I'd like us to be a bit more pragmatic.  Is minor and graceful
degradation for systems close to the trailing edge really so
unacceptably terrible that we have to bend over backwards to avoid it?

>> All the angst about CentOS falling off the end of our "supported build
>> platforms" list is not actually warranted by this series :)
>
> Using the term "angst" for the concerns of your fellows here is quite 
> cheeky. It's not about "angst", it's about a discussion that our support 
> policy might need to be adjusted if we do this step. So instead of writing 
> such sentences, I'd rather would like to see you posting a patch for 
> docs/about/build-platforms.rst for constructive further discussion instead.

The phrasing of this sentence was ill-advised.  If it caused offense, I
apologize.

On Fri, Feb 17, 2023 at 10:06:49AM +0100, Markus Armbruster wrote:
> Thomas Huth <thuth@redhat.com> writes:
> 
> > On 15/02/2023 20.05, Markus Armbruster wrote:
> >> The discussion under PATCH 6 makes me think there's a bit of confusion
> >> about the actual impact of dropping support for Python 3.6.  Possibly
> >> because it's spelled out in the commit message of PATCH 7.  Let me
> >> summarize it in one sentence:
> >> 
> >>      *** All supported host systems continue to work ***
> >> 
> >> Evidence: CI remains green.
> >
> > The CI remains green since one of the patches disabled the building of the 
> > docs on CentOS 8. That's not how I'd describe "continue to work", at least 
> > not in the same extend as before.
> 
> Thus the exception ...
> 
> >> On some supported host systems, different packages need to be installed.
> >> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> >> instead of 3.6.8.  Let me stress again: same repository, different
> >> package.  No downsides I can see.
> 
> ... right here:
> 
> >> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> >> version of Sphinx that works with Python 3.7 or newer.  This series
> >> proposes to simply stop building the docs there, unless the user
> >> provides a suitable version of Sphinx (which is easy enough with pip).
> >
> > I think we've all understood that. The thing that you obviously did not 
> > understood: This breaks our support statement.
> > I'm pretty sure that you could also build the whole QEMU suite successfully 
> > on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a 
> > newer version of GCC and some of the required libraries first. But that's 
> > not how we understand our support statement.
> >
> > Sure, you can argue that you can use "pip install" to get a newer version of 
> > Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is 
> > that really that much different from installing a newer version of GCC and 
> > libraries on an ancient distro that we do not officially support anymore? 
> > I'd say no. You also have to consider that not every build host has access 
> > to the internet, maybe some companies only have an internal mirror of the 
> > distro packages in their intranet (I remember some discussion about such a 
> > case in the past) - so while you were perfectly fine to build the whole of 
> > QEMU on a CentOS 8 there before this change, you could now not build parts 
> > of QEMU anymore there due to the missing possibility to run "pip install" 
> > without full internet connection.
> >
> > And sure, you can argue that it's "just" the documentation. But IMHO that's 
> > still an essential part of the QEMU build, and it used to work before, so it 
> > feels wrong to cut that now out. And also, if we start with the 
> > documentation now, what's next? If for example scripts/shaderinclude.py 
> > stops working with Python 3.6, do we then simply say: "Oh, it's fine, you 
> > can still build all the other targets that work without this script, just 
> > not the ones anymore that need it"?
> 
> My view on all this is a bit more pragmatic.
> 
> For a human developer, the difference between "dnf install
> python-sphinx" and "pip install sphinx" is, in my opinion, close to
> negligible.  Really no comparison to "git-clone GCC and bootstap it".
> You seem to disagree with that.

The command for installing the package is certainly not
significantly different.

What I think is lacking with pip is what happens after that
point - the ongoing management of the installed packages, or
rather than lack thereof.

If doing user installs, my $HOME/.local/lib ends up with multiple
orphaned previously installed versions. After each python version
upgrade, pip ceases to report about existance previously installed
bits at all, alone ever upgrade them.

Then there is the issue that pip will refuse to install a package,
if it already exists either from pip or the distro install, even
if the version that already exists doesn't satisfy the version
required. This would impact if the user has 'flake8'  N installed,
and QEMU wanted version N+1 installed, both using the same python
runtime version.

This is not the situation we're in with centos8/docutils, but
it is plausible in future once we decide we'll no longer rely
on exclusively on distro provided python packages, and instead
use pip if we ever want a newer version of something.


The python answer to this IIUC, is to use virtualenvs to isolate
everything. If we use keep the virtualenv under the QEMU build
root, then any time you clean your build root the virtualenv
content needs re-populating. At least pip has the sense to use
a cache, so $HOME/.cache/pip keeps a copy of everything you've
previously downloaded (presumably they age out the cache at
some time/size?), to avoid spending time re-loading all the
time. 


NB, I'm not saying the above is a blocker against use of pip.
Just that there are real differences between use of pip and
the distro packaging tools, beyond the initial install command.


If we're going to use pip, then using virtualenvs looks likely
to be very desirable to avoid conflicts between versions of
pkgs QEMU wants vs what the user might already have installed,
as well as to avoid leaving cruft behind in $HOME/.local/lib.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 17/02/2023 10.06, Markus Armbruster wrote:
> Thomas Huth <thuth@redhat.com> writes:
...
> My view on all this is a bit more pragmatic.
> 
> For a human developer, the difference between "dnf install
> python-sphinx" and "pip install sphinx" is, in my opinion, close to
> negligible.  Really no comparison to "git-clone GCC and bootstap it".
> You seem to disagree with that.

Honestly, being a Python ignorant, I completely messed up my system with 
"pip" already a couple of times, especially if the instructions forgot to 
tell me to use the "--user" switch. So yes, I tend to disagree ;-)

> For automated builds in general, and distro packaging in particular, the
> difference is real, and could even be a show stopper.  But who's
> packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
> builds are our own CI, where the difference is real, but hardly a show
> stopper.

If we've got the feeling that nobody out there really builds QEMU on older 
long-term distros anymore, then why the heck are we still trying to support 
this according to our support statement?

> But then I'd like us to be a bit more pragmatic.  Is minor and graceful
> degradation for systems close to the trailing edge really so
> unacceptably terrible that we have to bend over backwards to avoid it?

Let's just get our support statement adjusted - it was written with good 
intention originally, but apparently this is causing too much pain, so we 
should adjust it instead of suffering in upstream development.

>>> All the angst about CentOS falling off the end of our "supported build
>>> platforms" list is not actually warranted by this series :)
>>
>> Using the term "angst" for the concerns of your fellows here is quite
>> cheeky. It's not about "angst", it's about a discussion that our support
>> policy might need to be adjusted if we do this step. So instead of writing
>> such sentences, I'd rather would like to see you posting a patch for
>> docs/about/build-platforms.rst for constructive further discussion instead.
> 
> The phrasing of this sentence was ill-advised.  If it caused offense, I
> apologize.

Ok, thanks. And just to make it clear again: I certainly do not object 
dropping the support for Python 3.6 - I just want to make sure that we 
adjust our support statement if the current version is causing too much pain 
for us. Sorry if I got that across in the wrong way.

  Thomas

On Fri, 17 Feb 2023 at 09:56, Thomas Huth <thuth@redhat.com> wrote:
>
> On 17/02/2023 10.06, Markus Armbruster wrote:
> > Thomas Huth <thuth@redhat.com> writes:
> ...
> > My view on all this is a bit more pragmatic.
> >
> > For a human developer, the difference between "dnf install
> > python-sphinx" and "pip install sphinx" is, in my opinion, close to
> > negligible.  Really no comparison to "git-clone GCC and bootstap it".
> > You seem to disagree with that.
>
> Honestly, being a Python ignorant, I completely messed up my system with
> "pip" already a couple of times, especially if the instructions forgot to
> tell me to use the "--user" switch. So yes, I tend to disagree ;-)

Seconded. I trust my distro package manager and I know how it works,
and I know how to uninstall a package later if I want to revert what
I've done. I do not know or trust what the heck pip is doing or where it's
trying to install anything, because it's not a tool I habitually
use. I can't remember if I've managed to mess up the system with it,
but I've definitely had the experience of "install stuff with pip,
do a distro upgrade later, the pip installed stuff is all busted".

> > For automated builds in general, and distro packaging in particular, the
> > difference is real, and could even be a show stopper.  But who's
> > packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
> > builds are our own CI, where the difference is real, but hardly a show
> > stopper.
>
> If we've got the feeling that nobody out there really builds QEMU on older
> long-term distros anymore, then why the heck are we still trying to support
> this according to our support statement?

I don't think anybody is *packaging* new QEMU on an old distro.
I do think we have users who do ad-hoc from-source builds.

thanks
-- PMM

On Fri, Feb 17, 2023 at 03:37:16PM +0000, Peter Maydell wrote:
> On Fri, 17 Feb 2023 at 09:56, Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 17/02/2023 10.06, Markus Armbruster wrote:
> > > Thomas Huth <thuth@redhat.com> writes:
> > ...
> > > My view on all this is a bit more pragmatic.
> > >
> > > For a human developer, the difference between "dnf install
> > > python-sphinx" and "pip install sphinx" is, in my opinion, close to
> > > negligible.  Really no comparison to "git-clone GCC and bootstap it".
> > > You seem to disagree with that.
> >
> > Honestly, being a Python ignorant, I completely messed up my system with
> > "pip" already a couple of times, especially if the instructions forgot to
> > tell me to use the "--user" switch. So yes, I tend to disagree ;-)
> 
> Seconded. I trust my distro package manager and I know how it works,
> and I know how to uninstall a package later if I want to revert what
> I've done. I do not know or trust what the heck pip is doing or where it's
> trying to install anything, because it's not a tool I habitually
> use. I can't remember if I've managed to mess up the system with it,
> but I've definitely had the experience of "install stuff with pip,
> do a distro upgrade later, the pip installed stuff is all busted".
> 
> > > For automated builds in general, and distro packaging in particular, the
> > > difference is real, and could even be a show stopper.  But who's
> > > packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
> > > builds are our own CI, where the difference is real, but hardly a show
> > > stopper.
> >
> > If we've got the feeling that nobody out there really builds QEMU on older
> > long-term distros anymore, then why the heck are we still trying to support
> > this according to our support statement?
> 
> I don't think anybody is *packaging* new QEMU on an old distro.

I recall that at one time the openvz folks where packaging new QEMU on
RHEL-7 for a while after we had already dropped RHEL-7 as a target.
That's the trouble with enterprise distros, their usage sticks around
way longer than any of us would care to admit.

> I do think we have users who do ad-hoc from-source builds.

We'll certainly have contributors using it as a dev platform from
the corporate world.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Wed, 15 Feb 2023 at 19:05, Markus Armbruster <armbru@redhat.com> wrote:
>
> The discussion under PATCH 6 makes me think there's a bit of confusion
> about the actual impact of dropping support for Python 3.6.  Possibly
> because it's spelled out in the commit message of PATCH 7.  Let me
> summarize it in one sentence:
>
>     *** All supported host systems continue to work ***
>
> Evidence: CI remains green.
>
> On some supported host systems, different packages need to be installed.
> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> instead of 3.6.8.  Let me stress again: same repository, different
> package.  No downsides I can see.

Yes; I never had any issues with this part of it. If there was
a "Sphinx that also used that Python" in that repo, the answer
would be easy.

> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> version of Sphinx that works with Python 3.7 or newer.  This series
> proposes to simply stop building the docs there, unless the user
> provides a suitable version of Sphinx (which is easy enough with pip).
> That's PATCH 7.

Yes; this brings CentOS 8 down from "fully supported" to "kinda
supported but not for everything", which is less than ideal.
I think the level of not-idealness of that side of the scales
is probably clear enough. The difficulty I think for those who
haven't had their arms deep in QEMU's Python code is not having
the background info to be able to weigh up how heavy the other side
of the tradeoff scales is (since the naive "well, just keep writing
Python 3.6 for the moment" take is clearly wrong).

thanks
-- PMM

Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 15 Feb 2023 at 19:05, Markus Armbruster <armbru@redhat.com> wrote:
>>
>> The discussion under PATCH 6 makes me think there's a bit of confusion
>> about the actual impact of dropping support for Python 3.6.  Possibly
>> because it's spelled out in the commit message of PATCH 7.  Let me
>> summarize it in one sentence:
>>
>>     *** All supported host systems continue to work ***
>>
>> Evidence: CI remains green.
>>
>> On some supported host systems, different packages need to be installed.
>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>> instead of 3.6.8.  Let me stress again: same repository, different
>> package.  No downsides I can see.
>
> Yes; I never had any issues with this part of it. If there was
> a "Sphinx that also used that Python" in that repo, the answer
> would be easy.
>
>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>> version of Sphinx that works with Python 3.7 or newer.  This series
>> proposes to simply stop building the docs there, unless the user
>> provides a suitable version of Sphinx (which is easy enough with pip).
>> That's PATCH 7.
>
> Yes; this brings CentOS 8 down from "fully supported" to "kinda
> supported but not for everything", which is less than ideal.

I acknowledge there's a difference between "you need to dnf install
python-sphinx to be able to build docs" and "you need to pip install
sphinx to be able to build docs", and that the difference is negligible
in some scenarios, and a show stopper in others.  I wasn't fully aware
of the latter.

> I think the level of not-idealness of that side of the scales
> is probably clear enough. The difficulty I think for those who
> haven't had their arms deep in QEMU's Python code is not having
> the background info to be able to weigh up how heavy the other side
> of the tradeoff scales is (since the naive "well, just keep writing
> Python 3.6 for the moment" take is clearly wrong).

Fair point.  I hope the situation is more clear now.

On Thu, Feb 9, 2023 at 7:31 PM John Snow <jsnow@redhat.com> wrote:
>
> Howdy, this series increases our minimum python version to 3.7.
>
> CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
>     (All green!)
> GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37
>
> Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
> qemu.git within the week whether or not we take this series. I'd
> appreciate an "ACK" on those specifically. They're just riding along
> here because they make this series a bit nicer.
>
> Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.
>
> The motivation for this series is that Python 3.6 was EOL at the end of
> 2021; upstream tools are beginning to drop support for it, including
> setuptools, pylint, mypy, etc. As time goes by, it becomes more
> difficult to support and test against the full range of Python versions
> that QEMU supports. The closer we get to Python 3.12, the harder it will
> be to cover that full spread of versions.
>
> The qemu.qmp library and the avocado testing framework both have
> motivations for dropping 3.6 support, but are committed to not doing so
> until QEMU drops support.
>
> So, I'd like to talk about doing it.
>
> V2:
> - Added R-Bs to patch 1
> - Updated commit message for patch 7 with explicit version info
> - Added DO-NOT-MERGE to patch 5's title
> - Tested tests/vm/freebsd, netbsd, and openbsd in addition to full CI
>
> RFC:
>  - Patch 5 is just a proof-of-concept; we need to update lcitool instead.
>  - Cleber, I need to update your ansible scripts. How do I test them?
>
> Thanks!
> --js
>
> John Snow (7):
>   python: support pylint 2.16
>   python: drop pipenv

Hi, I've staged these first two patches to my Python branch.

(Kevin, Hanna; is that acceptable? I touch some iotests to do some
trivial linting whack-a-mole.)

--js

>   configure: Look for auxiliary Python installations
>   configure: Add nice hint to Python failure message
>   DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE
>   CI: Stop building docs on centos8
>   Python: Drop support for Python 3.6
>
>  docs/conf.py                                  |   4 +-
>  python/README.rst                             |   3 -
>  configure                                     |  40 +-
>  .gitlab-ci.d/buildtest.yml                    |   2 +-
>  .gitlab-ci.d/static_checks.yml                |   4 +-
>  python/.gitignore                             |   4 +-
>  python/Makefile                               |  57 ++-
>  python/Pipfile                                |  13 -
>  python/Pipfile.lock                           | 347 ------------------
>  python/qemu/qmp/protocol.py                   |   2 +-
>  python/qemu/qmp/qmp_client.py                 |   2 +-
>  python/qemu/utils/qemu_ga_client.py           |   6 +-
>  python/setup.cfg                              |  11 +-
>  python/tests/minreqs.txt                      |  45 +++
>  scripts/qapi/mypy.ini                         |   2 +-
>  tests/docker/dockerfiles/centos8.docker       |   1 +
>  tests/docker/dockerfiles/opensuse-leap.docker |   1 +
>  tests/docker/dockerfiles/python.docker        |   1 -
>  tests/qemu-iotests/iotests.py                 |   4 +-
>  .../tests/migrate-bitmaps-postcopy-test       |   2 +-
>  20 files changed, 135 insertions(+), 416 deletions(-)
>  delete mode 100644 python/Pipfile
>  delete mode 100644 python/Pipfile.lock
>  create mode 100644 python/tests/minreqs.txt
>
> --
> 2.39.0
>
>

Am 14.02.2023 um 19:35 hat John Snow geschrieben:
> On Thu, Feb 9, 2023 at 7:31 PM John Snow <jsnow@redhat.com> wrote:
> >
> > Howdy, this series increases our minimum python version to 3.7.
> >
> > CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
> >     (All green!)
> > GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37
> >
> > Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
> > qemu.git within the week whether or not we take this series. I'd
> > appreciate an "ACK" on those specifically. They're just riding along
> > here because they make this series a bit nicer.
> >
> > Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.
> >
> > The motivation for this series is that Python 3.6 was EOL at the end of
> > 2021; upstream tools are beginning to drop support for it, including
> > setuptools, pylint, mypy, etc. As time goes by, it becomes more
> > difficult to support and test against the full range of Python versions
> > that QEMU supports. The closer we get to Python 3.12, the harder it will
> > be to cover that full spread of versions.
> >
> > The qemu.qmp library and the avocado testing framework both have
> > motivations for dropping 3.6 support, but are committed to not doing so
> > until QEMU drops support.
> >
> > So, I'd like to talk about doing it.
> >
> > V2:
> > - Added R-Bs to patch 1
> > - Updated commit message for patch 7 with explicit version info
> > - Added DO-NOT-MERGE to patch 5's title
> > - Tested tests/vm/freebsd, netbsd, and openbsd in addition to full CI
> >
> > RFC:
> >  - Patch 5 is just a proof-of-concept; we need to update lcitool instead.
> >  - Cleber, I need to update your ansible scripts. How do I test them?
> >
> > Thanks!
> > --js
> >
> > John Snow (7):
> >   python: support pylint 2.16
> >   python: drop pipenv
> 
> Hi, I've staged these first two patches to my Python branch.
> 
> (Kevin, Hanna; is that acceptable? I touch some iotests to do some
> trivial linting whack-a-mole.)

Yes, of course.

Kevin

First, a plea.  Supporting 3.6 has made a few of us prisoners dragging
ball and chain.  So, please, *please* let us cut of these leg irons!

The series does not include follow-up cleanups.  Fine with me.

On Thu, Feb 09, 2023 at 07:31:40PM -0500, John Snow wrote:
> Howdy, this series increases our minimum python version to 3.7.

..snip...

So based on the discussion in this thread I see the following as the
key take aways:

 * Use of newer Meson feature

   We need newer meson in order to continue/finish the conversion
   of configure shell script into meson.build file logic. The newer
   meson has python 3.7 as a minimum required.

   The switch to meson was/is a significant strategic goal for
   improving QEMU build system maintainability which is thus
   highly desirable to continue without arbitrary delay.

 * Use of newer python parameter type hinting annotations

   Newer mypy supports a wider range of python type hinting
   annotations. While these ought to only impact the need for
   a new mypy for offline static analysis, in practice they
   also impact the runtime execution of the python code
   breaking compatibility with python 3.6. This unfortunate
   but out of our control.

   The wider use of type hinting in our python code is improving
   its maintainability, because it makes it significantly simpler
   for contributors to understand the behaviour of the code.


 * Use of newer python static analysis tools in general
 
   This applies to thinks like pylint, flak8, mypy (any more?)
   The issues reported by newer versions are not neccessarily
   always a strict superset of issues reports by older versions.
   Thus you can't assume that a passing result from new tools
   implies a passing result from the old tools. As such supporting
   all older versions is a cost/burden with little to no upside.

   With the exception of mypy, these aren't believed to have a
   direct bearing on the minimum python runtime version, but this
   would still technically be a deviation from our current support
   policy.


The first two points feel like pretty decent functional reasons to
justify an increase in the minimum python version. They will both
unlock the use of features that will have a tangible benefit to the
maintainability of QEMU in key areas.

If we do the first two points, then we can justify the 3rd point too
as having negligible additional downside.


The question is how to reconcile this with the platform support policy.
The policy has two goals. First to give us as maintainers a clear rule
as to when it is acceptable to increase minimum versions. The intent
is we don't need to debate the merits of each proposed change, just
validate that it follows our stated support policy. The second goal
is to give consumers of QEMU guidance as to what their platform has
to be able to provide in order to build and deploy QEMU.

The long life enterprise distros are the pain point in unlocking the
new of python features. So the proposal is that, at a minimum, we
augment the current policy with words to the effect that:

  * For long life cycle distributions, QEMU will follow normal
    policy for platform level dependancies / native code. For
    Python modules, QEMU may choose to require a newer versions
    than are available from the distribution package manager.

Potentially we could go further and reserve the right to
mandate newer python versions for any OS distro, because once
we have the facility for doing this with enterprise distros,
that shoudl just work anywhere. Perhaps start with targetted
exception at long life distros though. Either would unblock
our immediate needs wrt droppping python 3.6.

Finally, in terms of putting this into practice

  * If the OS platform doesn't provide the sufficiently new
    python modules, QEMU should default to downloading and
    installing them using pip, into a virtualenv under the
    QEMU build root so we don't impact other stuff the user
    might have in $HOME/.local/lib/python*. IIUC, the use
    of virtualenvs has been desired for a while already.

  * QEMU must provide a mechanism to disable any automatic
    downloading with pip, to allow users to pre-populate the
    addon modules if desired, to cope with scenarios where
    network access isn't available. Should be straightforward.

  * CI should continue to test all features, and be able
    to install the extra pieces with pip, if relevant for
    the distro. This should already be doable with the
    lcitool stuff Paolo has got working, which seemlessly
    uses pip on distros if needed.


So basically I'm suggesting we take a combination of John's series
here, and Paolo's series. John's last patch would change so that our
dockerfiles grab docutils with pip instead of disabling docs build
on centos8. Then look at introducing the use of virtualenvs for auto
installing any python mods the OS doesn't satisfy.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 2/17/23 12:37, Daniel P. Berrangé wrote:
> 
> So basically I'm suggesting we take a combination of John's series
> here, and Paolo's series. John's last patch would change so that our
> dockerfiles grab docutils with pip instead of disabling docs build
> on centos8. Then look at introducing the use of virtualenvs for auto
> installing any python mods the OS doesn't satisfy.

Sounds good.  I'll post a v3 of this series shortly, in the meanwhile 
I've already posted a suggested update to the support policy, but don't 
plan to drop support for Python 3.6 in 7.0 so that we can give a heads 
up in the release notes.

John is looking at building the virtualenv at configure time, which will 
have the characteristics that you detailed, so both the minimum version 
bump and the virtualenv change would happen in time for 7.1.

Paolo

On 17/02/2023 12.37, Daniel P. Berrangé wrote:
...
> The long life enterprise distros are the pain point in unlocking the
> new of python features. So the proposal is that, at a minimum, we
> augment the current policy with words to the effect that:
> 
>    * For long life cycle distributions, QEMU will follow normal
>      policy for platform level dependancies / native code. For
>      Python modules, QEMU may choose to require a newer versions
>      than are available from the distribution package manager.

Sounds reasonable to me. But I think we still should also add a sentence 
where we limit the total amount of time that we promise to support a 
long-term distro. Otherwise we'll also get problems with other way too 
backlevel native code dependencies at one point in time.

I just sent a patch for discussion, shortly before I noticed your mail here. 
Feel free to grab the ideas from there into your patch (if you're planning 
to send one), or let me know if I should try to include the Python-related 
sentences in mine.

  Thomas

On Fri, Feb 17, 2023 at 02:46:53PM +0100, Thomas Huth wrote:
> On 17/02/2023 12.37, Daniel P. Berrangé wrote:
> ...
> > The long life enterprise distros are the pain point in unlocking the
> > new of python features. So the proposal is that, at a minimum, we
> > augment the current policy with words to the effect that:
> > 
> >    * For long life cycle distributions, QEMU will follow normal
> >      policy for platform level dependancies / native code. For
> >      Python modules, QEMU may choose to require a newer versions
> >      than are available from the distribution package manager.
> 
> Sounds reasonable to me. But I think we still should also add a sentence
> where we limit the total amount of time that we promise to support a
> long-term distro. Otherwise we'll also get problems with other way too
> backlevel native code dependencies at one point in time.

Oh sure, having something time related mention makes sense.

> I just sent a patch for discussion, shortly before I noticed your mail here.
> Feel free to grab the ideas from there into your patch (if you're planning
> to send one), or let me know if I should try to include the Python-related
> sentences in mine.

I wasn't planning to imminently send a change. Especially now that
both yourself an Paolo have proposed similar competing additions,
lets just iterate on one of those 2 proposals.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|