CentOS 8 does not ship with a sphinx new enough for our purposes (It
necessarily uses Python 3.6), so drop this from this build. We can
resume building docs on CentOS 9 if we wish, but we also currently test
and build docs on Fedora, Ubuntu, Alpine and Debian.
Signed-off-by: John Snow <jsnow@redhat.com>
---
.gitlab-ci.d/buildtest.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 0aa149a3524..0eb7f6606d4 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -166,7 +166,7 @@ build-system-centos:
variables:
IMAGE: centos8
CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
- --enable-modules --enable-trace-backends=dtrace --enable-docs
+ --enable-modules --enable-trace-backends=dtrace
--enable-vfio-user-server
TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
--
2.39.0On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote: > > CentOS 8 does not ship with a sphinx new enough for our purposes (It > necessarily uses Python 3.6), so drop this from this build. We can > resume building docs on CentOS 9 if we wish, but we also currently test > and build docs on Fedora, Ubuntu, Alpine and Debian. This confuses me. We work fine with Python 3.6 today. Either: * CentOS 8 has fallen off the end of our "supported build platforms" list -- if so, we don't need to be CI'ing anything on it. * CentOS 8 is still a supported platform -- in this case building the docs is something we need to continue to support, and we can't drop Python 3.6 until all of our supported build platforms have a newer Python available. thanks -- PMM
On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote: > On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote: > > > > CentOS 8 does not ship with a sphinx new enough for our purposes (It > > necessarily uses Python 3.6), so drop this from this build. We can > > resume building docs on CentOS 9 if we wish, but we also currently test > > and build docs on Fedora, Ubuntu, Alpine and Debian. > > This confuses me. We work fine with Python 3.6 today. > That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments. Our testing matrix for Python is novel and thorough enough that it's revealed several bugs in other downstream Python distributions for Debian and Fedora, and dozens of bugs for the linters themselves. I'm concerned that when 3.7 is EOL'd in just a few months that the support and testing gap is going to get even uglier. In addition, Avocado framework has been asking me to drop 3.6 so they can follow suit, for much of the same reasons - maintenance cost. The typing work we've been pursuing for our testing and qapi libraries would also be greatly helped by dropping 3.6, but it's really the maintenance argument that's driving this. Anyway, just trying to explain that it's not just a frivolous thing I'm doing for the sake of itself. Either: > * CentOS 8 has fallen off the end of our "supported build platforms" > list -- if so, we don't need to be CI'ing anything on it. > * CentOS 8 is still a supported platform -- in this case building > the docs is something we need to continue to support, and we > can't drop Python 3.6 until all of our supported build > platforms have a newer Python available. > The argument I'm making is: - CentOS 8 is a supported build platform - All platforms *do* have a Python modern enough to allow us to drop 3.6 - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though - You expressed a preference to me in the past to NOT use a pip installed version of sphinx in preference to the system version in "configure" - It's still possible to build docs on CentOS 8 after this patchset, you just need a pip version. - We've used the justification that our build platform promise does not necessarily extend to docs and tests in the past. - So just skip docs building for CentOS 8, only in the CI. If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want to go back to discussing the possibility of using sphinx versions from pip. Paolo mentioned wanting to do similar in reply to another patch on this series (#5) to preserve docs building on the platform. > thanks > -- PMM > >
On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote: > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote: >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote: >> This confuses me. We work fine with Python 3.6 today. > > > That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments. Something somewhere seems kind of out-of-sync here. Either: * Python is deprecating old versions too quickly and dropping support for them too fast * CentOS is retaining old versions of Python when it needs to ship newer ones * Our policy for what distros we support is overly lax and causing us to try to support ancient platforms that we shouldn't be trying to support because "use the system version of foo" should not be a big deal -- it's not like we're trying to support decades-old hosts here: Centos 8 was released in 2019, which is less than five years ago. > The argument I'm making is: > > - CentOS 8 is a supported build platform > - All platforms *do* have a Python modern enough to allow us to drop 3.6 > - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though > - You expressed a preference to me in the past to NOT use a pip installed version of sphinx in preference to the system version in "configure" > - It's still possible to build docs on CentOS 8 after this patchset, you just need a pip version. > - We've used the justification that our build platform promise does not necessarily extend to docs and tests in the past. > - So just skip docs building for CentOS 8, only in the CI. > > If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want to go back to discussing the possibility of using sphinx versions from pip. If Python 3.6 is EOL, shouldn't CentOS have shipped an updated version of Sphinx to go with their updated Python ? I'm not super-happy about dropping the docs build requirement, but I guess it's probably the least-worst choice. thanks -- PMM
On Fri, Feb 10, 2023, 11:32 AM Peter Maydell <peter.maydell@linaro.org> wrote: > On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote: > > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> > wrote: > >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote: > >> This confuses me. We work fine with Python 3.6 today. > > > > > > That won't last - Many tools such as mypy, pylint and flake8 which I use > to manage our python codebase have been dropping support for 3.6 and I've > had to implement an increasing number of workarounds to help keep it > possible to test 3.6 via CI while also ensuring our newest platforms work > as dev environments. > > Something somewhere seems kind of out-of-sync here. Either: > * Python is deprecating old versions too quickly and > dropping support for them too fast > I think this is true. The Python upstream ecosystem moves extremely fast and breaks backwards compatibility for tooling often. It is sometimes tough to mediate that flow rate difference. Part of the reason I added that "check-tox" CI test that is currently optional to run is specifically so it can find problems with cutting edge tooling - and it happens often. (It seems like every freeze, I find out a new upstream version of something breaks the testing, like clockwork.) On the other hand, Python 3.6 did launch some six years ago - 2016-12-22. It's probably reasonable most cutting edge tooling for f37/f38 no longer supports developing for a version this old. I can't deny that the Python package ecosystem is too cavalier about dropping for support for things too quickly, though. * CentOS is retaining old versions of Python when it needs to > ship newer ones > I'm not clear on the exact policies for Stream 8/9, but in our traditional releases we'd not ever upgrade this - we'd only do micro updates and backports. Instead, Stream 8/9 (and OpenSUSE too) offer optional Python versions that install side-by-side. So sphinx (as packaged as an rpm) continues to use the platform Python - 3.6 - but you *can* launch the "system sphinx" with your newer Python interpreter - if the version of sphinx shipped happens to cope with a newer interpreter. (If you run `python3.7 -m sphinx`, for example - it might work, but only if your rpm-packaged sphinx isn't using any features of Python that were dropped in 3.7. I'd wager it probably does work most of the time.) * Our policy for what distros we support is overly lax > and causing us to try to support ancient platforms > that we shouldn't be trying to support > Possibly. Part of the idea behind RHEL8/CentOS8 is that you'll get security fixes, but it should otherwise be a stable platform for production. Supporting this as a development environment for cutting edge versions does feel like a mismatch in visions at times. RHEL8 launched 2018-11-14, so is it reasonable to expect that cutting edge versions of QEMU 4 years in the future will build on it no problem? Meanwhile, OpenSUSE 15.4 released just eight months ago and still ships Python 3.6 as its default/platform Python. They don't ship a distro with a modern Python as default at all... Even the upcoming 15.5 release still appears to use Python 3.6 as the platform default. (Like RHEL, they offer optional additional interpreters if you want to run upstream software written within the last two years.) At least QEMU's stated support for RHEL8 runs out in another year or so. I wonder what OpenSUSE is planning to do. > > > because "use the system version of foo" should not be a big > deal -- it's not like we're trying to support decades-old > hosts here: Centos 8 was released in 2019, which is less than > five years ago. > Yeah, I have nothing cute to say to this. It's unfortunate that it's hard to support RHEL8 and Fedora 37 simultaneously as development environments for our devs. I do take great pains to reduce disruptions to everyone as I keep the python code afloat. I hope I have been successful these past few years. However, there are packaged versions of modern Python available for any platform we support that otherwise defaults to 3.6, so it's not a huge affair to continue supporting these older platforms. It'll just be as if you need one extra dependency on those systems. (Maybe two if you want to build docs.) I think that trade-off is fair; as it's only a one-time expense to developers still working on older platforms. > > The argument I'm making is: > > > > - CentOS 8 is a supported build platform > > - All platforms *do* have a Python modern enough to allow us to drop 3.6 > > - CentOS's repo version of sphinx is hardcoded to use the older 3.6, > though > > - You expressed a preference to me in the past to NOT use a pip > installed version of sphinx in preference to the system version in > "configure" > > - It's still possible to build docs on CentOS 8 after this patchset, you > just need a pip version. > > - We've used the justification that our build platform promise does not > necessarily extend to docs and tests in the past. > > - So just skip docs building for CentOS 8, only in the CI. > > > > If you believe docs in CI for CentOS 8 is a hard deal breaker, then I > want to go back to discussing the possibility of using sphinx versions from > pip. > > If Python 3.6 is EOL, shouldn't CentOS have shipped an updated > version of Sphinx to go with their updated Python ? > It's EOL upstream, but I suppose the RHEL/CentOS view is that we'll support it with our own security fixes/backports past its upstream death. I don't know the exact versioning policies, but generally they never do a version bump. I don't expect them to ever upgrade the platform default python from 3.6, because it will likely involve hundreds of version bumps to other packages as a result. > I'm not super-happy about dropping the docs build requirement, > but I guess it's probably the least-worst choice. > I think Paolo is advocating for allowing the build to use a pip-installed version of sphinx, and configuring our docker containers to set up that environment. We'd keep CentOS 8 building docs just fine as a result. It would mean that individual contributors that wanted to build docs on CentOS 8 would need to install a sphinx version with pip, though. (The problem with just allowing sphinx to be a black box and continuing to happily use the 3.6-based versions is that we are using QAPIDoc extensions from our own codebase, which would lock those to 3.6. A big motivator for Markus is dropping some 3.6 kludges we're carrying for qapi, so I looked to the opposite solution - nudging Sphinx forward instead.) > thanks > -- PMM > Sorry for the essay! --js >
On Fri, 10 Feb 2023 at 17:55, John Snow <jsnow@redhat.com> wrote: > (The problem with just allowing sphinx to be a black box and > continuing to happily use the 3.6-based versions is that we are > using QAPIDoc extensions from our own codebase, which would lock > those to 3.6. A big motivator for Markus is dropping some 3.6 > kludges we're carrying for qapi, so I looked to the opposite > solution - nudging Sphinx forward instead.) Mmm. Where on the pain spectrum does "allow python 3.8 because CentosOS ships that, except that where our python code gets run via Sphinx that part of the codebase must retain 3.6 compatibility" sit ? thanks -- PMM
Il ven 10 feb 2023, 19:09 Peter Maydell <peter.maydell@linaro.org> ha scritto: > On Fri, 10 Feb 2023 at 17:55, John Snow <jsnow@redhat.com> wrote: > > (The problem with just allowing sphinx to be a black box and > > continuing to happily use the 3.6-based versions is that we are > > using QAPIDoc extensions from our own codebase, which would lock > > those to 3.6. A big motivator for Markus is dropping some 3.6 > > kludges we're carrying for qapi, so I looked to the opposite > > solution - nudging Sphinx forward instead.) > > Mmm. Where on the pain spectrum does "allow python 3.8 > because CentosOS ships that, except that where our python > code gets run via Sphinx that part of the codebase must > retain 3.6 compatibility" sit ? > As far as the build system is concerned no changes are required, however Python 3.7 support was requested by Markus for scripts/qapi/ for quite some time. So it would not be painful to implement but it would remove most of the benefits of dropping support for 3.6. Paolo > thanks > -- PMM > >
On Fri, Feb 10, 2023 at 04:32:19PM +0000, Peter Maydell wrote: > On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote: > > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote: > >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote: > >> This confuses me. We work fine with Python 3.6 today. > > > > > > That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments. > > Something somewhere seems kind of out-of-sync here. Either: > * Python is deprecating old versions too quickly and > dropping support for them too fast Nope, they're fine in declaring EOL whenever they like. There's no requirement for upstreams to support arbitrary old versions for any length of time. > * CentOS is retaining old versions of Python when it needs to > ship newer ones It is also totally OK for an distro to ship and support software versions which are EOL upstream. In fact for enterprise distros you can generally assume that *all* the software versions they ship are probably EOL or nearly so. The main value of enterprise distros is that they provide long term support, where upstreams are not willing to. > * Our policy for what distros we support is overly lax > and causing us to try to support ancient platforms > that we shouldn't be trying to support I don't think so. Users of distros are not anywhere near as aggressive at updating their installations as users are. The number of users of RHEL-8 will dwarf that of RHEL-9 by a large margin for a decent amount of time yet. The challenge here is that upstream developers want to use shiny new features from latest upstream, and at the same time don't want to keep back compat with older versions that users see in their real world deployed distros, because that is a burden. Ideally upstream developers would never care about old versions and only ever target the very latest upstream. Meanwhile for users, apps would ideally support every version of every OS that's ever existed. Somewhere in the middle we have to strike a bargain that spreads the pain fairly between the two groups, accepting that this is going to be a burden for both to some degre. Our distro support policy is a decent attempt at doing that IMHO and has worked pretty well IMHO. We don't have to drop python 3.6. It is a choice because of a desire to be able to use some shiny new python features without caring about back compat. Similarly we don't have to use the new meson which drops support for python 3.6, it is again a choice because we want to rely on some new meson features. QEMU could easily carry on supporting python 3.6 until the affected distros drop out ofo our support matrix, but we would have to opt out of using certain new features, or put more effort into back compat. Personally I'm still on the side of carrying on supporting 3.6 per our distro support policy, but if broad consensus is to drop 3.6 I'm not going push back anymore. > because "use the system version of foo" should not be a big > deal -- it's not like we're trying to support decades-old > hosts here: Centos 8 was released in 2019, which is less than > five years ago. Yeah, it isn't very old at all, and in terms of deployments will dominate over CentOS/RHEL 9 users. > > The argument I'm making is: > > > > - CentOS 8 is a supported build platform > > - All platforms *do* have a Python modern enough to allow us to drop 3.6 > > - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though > > - You expressed a preference to me in the past to NOT use a pip installed version of sphinx in preference to the system version in "configure" > > - It's still possible to build docs on CentOS 8 after this patchset, you just need a pip version. > > - We've used the justification that our build platform promise does not necessarily extend to docs and tests in the past. > > - So just skip docs building for CentOS 8, only in the CI. > > > > If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want to go back to discussing the possibility of using sphinx versions from pip. > > If Python 3.6 is EOL, shouldn't CentOS have shipped an updated > version of Sphinx to go with their updated Python ? CentOS isn't dropping python 3.6 support. It will exist for the lifetime of CentOS 8. It just decided to also ship some newer python versions as an opt-in. These newer python versions are just the minimal runtime though. CentOS is not shipping all the add-on python modules it has with 3.6. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé <berrange@redhat.com> writes: [...] > We don't have to drop python 3.6. It is a choice because > of a desire to be able to use some shiny new python > features without caring about back compat. I read this on Friday, and decided to let it sit until after the weekend. Well, it's now Tuesday, and to be frank, it's still as offensively flippant as it was on Friday. It shows either ignorance of or cavalier disregard for the sheer amount of work some of us have had to put into keeping old versions of Python viable. The latter would be quite unlike you, so it must be the former. John has sunk *man-months* into keeping old versions of Python viable. I've put in a lot less myself, but still enough to be painfully aware of it. I figure Cleber and Beraldo are somewhere in between. Insinuating John's proposal is motivated by "a desire to be able to use some shiny new python features without caring about back compat" disrespects all this work. We should have a sober discussion on which versions of Python to work with, and the tradeoffs involved. But before I engage in that, I insist on resetting the frame: no, this is not about shiny, new Python features. It is about stopping the bleeding. It is about reducing what feels more and more like bullshit work to me, so we can actually accomplish stuff that matters. And let's give the people who have been doing the actual work the benefit of the doubt. [...]
On Tue, Feb 14, 2023 at 08:40:20AM +0100, Markus Armbruster wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > [...] > > > We don't have to drop python 3.6. It is a choice because > > of a desire to be able to use some shiny new python > > features without caring about back compat. > > I read this on Friday, and decided to let it sit until after the > weekend. Well, it's now Tuesday, and to be frank, it's still as > offensively flippant as it was on Friday. It shows either ignorance of > or cavalier disregard for the sheer amount of work some of us have had > to put into keeping old versions of Python viable. > > The latter would be quite unlike you, so it must be the former. I'm sorry, I don't mean it to be offensive. I'm genuinely not seeing from the descriptions in the series what the functional benefits are from dropping 3.6. > John has sunk *man-months* into keeping old versions of Python viable. > I've put in a lot less myself, but still enough to be painfully aware of > it. I figure Cleber and Beraldo are somewhere in between > > Insinuating John's proposal is motivated by "a desire to be able to use > some shiny new python features without caring about back compat" > disrespects all this work. I'm writing my comments based on what is described in the cover letter as the motivations for the change: [quote] The motivation for this series is that Python 3.6 was EOL at the end of 2021; upstream tools are beginning to drop support for it, including setuptools, pylint, mypy, etc. As time goes by, it becomes more difficult to support and test against the full range of Python versions that QEMU supports. The closer we get to Python 3.12, the harder it will be to cover that full spread of versions. [/quote] this is all about new/eol versions of software upstream, and I don't think that's a justification. QEMU explicitly aims to use distro provided versions and upstream EOL status is not relevant in that context. Even if using "pip" to install it is possible to limit yourself to upstream releases which still support 3.6. There is the separate issue of Meson dropping python 3.6 which motivates Paolo's series. Again though, we don't have to increase our minimum meson version, because meson is working today. It is our choice to to increase it to use latest available meson features. At some point we can decide what we have is good enough and we don't have to keep chasing the latest features. Maybe we're not there yet, but we should think about when that would be. [quote] The qemu.qmp library and the avocado testing framework both have motivations for dropping 3.6 support, but are committed to not doing so until QEMU drops support. [/quote] I suspect that this is more of a driver for the drop of 3.6, but I don't see any details. IOW overall justification come across as wanting to use new features, and follow upstream EOL, without any real detail of what we're going to gain from a functional POV. > We should have a sober discussion on which versions of Python to work > with, and the tradeoffs involved. But before I engage in that, I insist > on resetting the frame: no, this is not about shiny, new Python > features. It is about stopping the bleeding. It is about reducing what > feels more and more like bullshit work to me, so we can actually > accomplish stuff that matters. Every applications developer chooses an arbitrary cut off points for minimum software versions, depending on their particular needs. With our support policy we tried to express a reasonable tradeoff between keeping back compat, and being able to adopt new features. Obviously that tradeoff is not currently looking acceptable on the python side, but it is not clear why that is ? Can someone simply explain what we wil see as the benefit from dropping 3.6 / adopting 3.7 as the baseline ? With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Tue, Feb 14, 2023 at 08:40:20AM +0100, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>> [...]
>>
>> > We don't have to drop python 3.6. It is a choice because
>> > of a desire to be able to use some shiny new python
>> > features without caring about back compat.
>>
>> I read this on Friday, and decided to let it sit until after the
>> weekend. Well, it's now Tuesday, and to be frank, it's still as
>> offensively flippant as it was on Friday. It shows either ignorance of
>> or cavalier disregard for the sheer amount of work some of us have had
>> to put into keeping old versions of Python viable.
>>
>> The latter would be quite unlike you, so it must be the former.
>
> I'm sorry, I don't mean it to be offensive. I'm genuinely not seeing
> from the descriptions in the series what the functional benefits are
> from dropping 3.6.
Apology accepted, and point well taken.
>> John has sunk *man-months* into keeping old versions of Python viable.
>> I've put in a lot less myself, but still enough to be painfully aware of
>> it. I figure Cleber and Beraldo are somewhere in between
>>
>> Insinuating John's proposal is motivated by "a desire to be able to use
>> some shiny new python features without caring about back compat"
>> disrespects all this work.
>
> I'm writing my comments based on what is described in the cover letter
> as the motivations for the change:
>
> [quote]
> The motivation for this series is that Python 3.6 was EOL at the end of
> 2021; upstream tools are beginning to drop support for it, including
> setuptools, pylint, mypy, etc. As time goes by, it becomes more
> difficult to support and test against the full range of Python versions
> that QEMU supports. The closer we get to Python 3.12, the harder it will
> be to cover that full spread of versions.
> [/quote]
>
> this is all about new/eol versions of software upstream, and I don't
> think that's a justification. QEMU explicitly aims to use distro provided
> versions and upstream EOL status is not relevant in that context. Even
> if using "pip" to install it is possible to limit yourself to upstream
> releases which still support 3.6.
>
> There is the separate issue of Meson dropping python 3.6 which motivates
> Paolo's series. Again though, we don't have to increase our minimum meson
> version, because meson is working today. It is our choice to to increase
> it to use latest available meson features. At some point we can decide
> what we have is good enough and we don't have to keep chasing the latest
> features. Maybe we're not there yet, but we should think about when that
> would be.
>
> [quote]
> The qemu.qmp library and the avocado testing framework both have
> motivations for dropping 3.6 support, but are committed to not doing so
> until QEMU drops support.
> [/quote]
>
> I suspect that this is more of a driver for the drop of 3.6, but I
> don't see any details.
>
> IOW overall justification come across as wanting to use new features,
> and follow upstream EOL, without any real detail of what we're going
> to gain from a functional POV.
I think what we gain is there: "As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions." In other words, the gain is
"we make something that's hard and getting harder easier".
What isn't in the cover letter, only in commit message 6+7/7, i.e. this
patch and the next one, is what we pay for it: basically nothing (next
patch's commit message) except for the issue of Sphinx in CentOS 8 (this
patch's commit message). Putting at least a summary it in the cover
letter would have been better.
John did explain this again and in more detail in reply to Peter's "This
confuses me. We work fine with Python 3.6 today." Quote:
That won't last - Many tools such as mypy, pylint and flake8 which I use to
manage our python codebase have been dropping support for 3.6 and I've had
to implement an increasing number of workarounds to help keep it possible
to test 3.6 via CI while also ensuring our newest platforms work as dev
environments.
Our testing matrix for Python is novel and thorough enough that it's
revealed several bugs in other downstream Python distributions for Debian
and Fedora, and dozens of bugs for the linters themselves.
I'm concerned that when 3.7 is EOL'd in just a few months that the support
and testing gap is going to get even uglier.
[...]
The argument I'm making is:
- CentOS 8 is a supported build platform
- All platforms *do* have a Python modern enough to allow us to drop 3.6
- CentOS's repo version of sphinx is hardcoded to use the older 3.6, though
- You expressed a preference to me in the past to NOT use a pip installed
version of sphinx in preference to the system version in "configure"
- It's still possible to build docs on CentOS 8 after this patchset, you
just need a pip version.
- We've used the justification that our build platform promise does not
necessarily extend to docs and tests in the past.
- So just skip docs building for CentOS 8, only in the CI.
If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want
to go back to discussing the possibility of using sphinx versions from pip.
You even quoted most of it in your message :)
The second paragraph is evidence we've been doing something basically
nobody else does.
>> We should have a sober discussion on which versions of Python to work
>> with, and the tradeoffs involved. But before I engage in that, I insist
>> on resetting the frame: no, this is not about shiny, new Python
>> features. It is about stopping the bleeding. It is about reducing what
>> feels more and more like bullshit work to me, so we can actually
>> accomplish stuff that matters.
>
> Every applications developer chooses an arbitrary cut off points for
> minimum software versions, depending on their particular needs. With
> our support policy we tried to express a reasonable tradeoff between
> keeping back compat, and being able to adopt new features.
>
> Obviously that tradeoff is not currently looking acceptable on the
> python side, but it is not clear why that is ?
>
> Can someone simply explain what we wil see as the benefit from dropping
> 3.6 / adopting 3.7 as the baseline ?
Let me start with the costs, then move on to the benefits.
Every single supported build platform already provides Python >= 3.7,
and all the Python tools we use continue to work with it, with one
exception: Sphinx on CentOS 8.
Solutions for this issue proposed so far:
1. Do nothing. To build documentation on CentOS 8, you have to pip
install a working version of Sphinx, which is easy enough. As long
as we don't in CI, we have to disable building docs there (this
patch).
2. Have the build system pip install it for you. A bit of a paradigm
shift: so far our build system doesn't install anything.
3. Split the Python dependency for Sphinx off the general one. On
CentOS 8, we'd use 3.6 to run Sphinx, and 3.7+ for everything else.
On all other systems, we'd use the same 3.7+ for everything.
So the cost is our choice of "you have to pip install Sphinx to build
docs on CentOS 8" / "build system pip installs Sphinx" / "special Python
dependency for Sphinx". Feels quite modest to me.
The main benefit is we reduce the bleeding caused by trying to support
Python all the way back to 3.6 when basically nobody else does. Linters
in particular. How much of a benefit is this? I'd like you all to
trust John it's plenty big enough to justify the (modest!) cost. He's
been doing much of the bleeding, after all.
New features provided by 3.7 will be nice to have (especially for
typing), but that's not why we're doing this.
On Tue, Feb 14, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > [quote] > The motivation for this series is that Python 3.6 was EOL at the end of > 2021; upstream tools are beginning to drop support for it, including > setuptools, pylint, mypy, etc. As time goes by, it becomes more > difficult to support and test against the full range of Python versions > that QEMU supports. The closer we get to Python 3.12, the harder it will > be to cover that full spread of versions. > [/quote] > > this is all about new/eol versions of software upstream, and I don't > think that's a justification. QEMU explicitly aims to use distro provided > versions and upstream EOL status is not relevant in that context. Even > if using "pip" to install it is possible to limit yourself to upstream > releases which still support 3.6. > > There is the separate issue of Meson dropping python 3.6 which motivates > Paolo's series. Again though, we don't have to increase our minimum meson > version, because meson is working today. It is our choice to to increase > it to use latest available meson features. At some point we can decide > what we have is good enough and we don't have to keep chasing the latest > features. Maybe we're not there yet, but we should think about when that > would be. In the case of Meson, the main advantage is moving _all_ of the emulator configury out of the configure script. This requires add_global_dependencies which was added in 0.63. So in that case it is indeed mostly about shiny new features and it's not absolutely necessary. In the case of Python the issue is not the interpreter per se, though there are a couple new feature in Python 3.7 that are quite nice (for example improved data classes[1] or context variables[2]). The main problem as far as I understood (and have seen in my experience) is linting tools. New versions fix bugs that caused false positives, but also become more strict at the same time. The newer versions at the same time are very quick at dropping support for old versions of Python; while older versions sometimes throw deprecation warnings on new versions of Python. This makes it very hard to support a single version of, say, mypy that works on all versions from RHEL8 and SLE15 to Fedora 38 and Ubuntu 23.04. [1] https://peps.python.org/pep-0557/ [2] https://peps.python.org/pep-0567/ In fact this issue is the reason why RHEL9 does not package any of these tools and does not run them as part of building RPMs even though in principle it would be a good idea; it's too much of a pain to have a single version that works across all the packages in the distribution. Regarding your other suggestion: > * For non-native library/applications dependancies we aim > to support only the most recent distro version. Users > of older distros may need to dynamically fetch newer > deps. I think this is a good idea, but one issue with "only supporting the most recent distro version" is SUSE. While the most recent version of SLE is about 5 years old, there is no next version in sight---SUSE instead is working on their "Adaptable Linux Platform", but it's still in the prototype stage[3]. So alternatively we could put a 4 or 5 year cutoff after which you need to fetch newer deps. Considering the delays between freeze and release of distros like RHEL or SLE, in practice we would probably keep Python versions supported for 6-7 years. A 4 year cutoff in practice means that we would be able to drop Python 3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE is already over the threshold). In practice this means waiting until next April for John/Markus/myself, which I think is fair. Note that at least for now Meson need not to follow this rule; it is distributed with QEMU because configure must execute it before Make sets up the Python venv. This may change in the future of course, depending on the direction that the configure script takes with respect to setting up the venv, but I wouldn't hold my breath. However, the minimum required version of Meson of course must respect the oldest supported version of Python. Paolo [3] https://www.suse.com/c/the-first-prototype-of-adaptable-linux-platform-is-live/
On Tue, Feb 14, 2023 at 03:03:54PM +0100, Paolo Bonzini wrote: > On Tue, Feb 14, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > [quote] > > The motivation for this series is that Python 3.6 was EOL at the end of > > 2021; upstream tools are beginning to drop support for it, including > > setuptools, pylint, mypy, etc. As time goes by, it becomes more > > difficult to support and test against the full range of Python versions > > that QEMU supports. The closer we get to Python 3.12, the harder it will > > be to cover that full spread of versions. > > [/quote] > > > > this is all about new/eol versions of software upstream, and I don't > > think that's a justification. QEMU explicitly aims to use distro provided > > versions and upstream EOL status is not relevant in that context. Even > > if using "pip" to install it is possible to limit yourself to upstream > > releases which still support 3.6. > > > > There is the separate issue of Meson dropping python 3.6 which motivates > > Paolo's series. Again though, we don't have to increase our minimum meson > > version, because meson is working today. It is our choice to to increase > > it to use latest available meson features. At some point we can decide > > what we have is good enough and we don't have to keep chasing the latest > > features. Maybe we're not there yet, but we should think about when that > > would be. > > In the case of Meson, the main advantage is moving _all_ of the > emulator configury out of the configure script. This requires > add_global_dependencies which was added in 0.63. So in that case it > is indeed mostly about shiny new features and it's not absolutely > necessary. I forgot to mention in my previous reply, I feel like the ability to finish the configure -> meson conversion is a pretty compelling motivation to adopt the new meson. The hybrid configure/meson state we've been in has worked better than I expected it would at first, but none the less, the more we can get out of configure the better it will be for ongoing maint burden. So yes, its shiny new features, but they're pretty compelling features as they allow us to finish the job we started. We've clear precedent all over QEMU codebase that half-finished conversions harm our ability to maintain the project. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: > In the case of Python the issue is not the interpreter per se, though > there are a couple new feature in Python 3.7 that are quite nice (for > example improved data classes[1] or context variables[2]). The main > problem as far as I understood (and have seen in my experience) is > linting tools. New versions fix bugs that caused false positives, but > also become more strict at the same time. The newer versions at the > same time are very quick at dropping support for old versions of > Python; while older versions sometimes throw deprecation warnings on > new versions of Python. This makes it very hard to support a single > version of, say, mypy that works on all versions from RHEL8 and SLE15 > to Fedora 38 and Ubuntu 23.04. Why do we have to support a single version of mypy? What is wrong with running an old mypy version with old Python version, and a newer mypy with newer Python versions? Sure, they will complain about different things, but it doesn't feel that different from supporting multiple C compilers in various versions. Kevin
On Tue, Feb 14, 2023 at 12:26 PM Kevin Wolf <kwolf@redhat.com> wrote: > > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: > > In the case of Python the issue is not the interpreter per se, though > > there are a couple new feature in Python 3.7 that are quite nice (for > > example improved data classes[1] or context variables[2]). The main > > problem as far as I understood (and have seen in my experience) is > > linting tools. New versions fix bugs that caused false positives, but > > also become more strict at the same time. The newer versions at the > > same time are very quick at dropping support for old versions of > > Python; while older versions sometimes throw deprecation warnings on > > new versions of Python. This makes it very hard to support a single > > version of, say, mypy that works on all versions from RHEL8 and SLE15 > > to Fedora 38 and Ubuntu 23.04. > > Why do we have to support a single version of mypy? What is wrong with > running an old mypy version with old Python version, and a newer mypy > with newer Python versions? Well, the problem is, ... > > Sure, they will complain about different things, but it doesn't feel > that different from supporting multiple C compilers in various versions. ...well, it's this. The first dimension of the test matrix is the version of mypy itself. The second dimension of the test matrix is the version of Python mypy runs under. A given version of mypy can run under multiple versions of Python and may indeed have different behaviors under each. The third dimension of this test matrix is the version(s) of Python our code is targeting; for instance we configure mypy to understand that we require Python 3.6. Trying to cast the net wide on *all of these* gets tough; the very latest versions of mypy don't support 3.6 at all, so you'll have cases where the mypy that just so happens to come with your Fedora installation just won't work properly with our code anymore, which has to test under 3.6 appropriately. In general, the majority of python package upstreams I am aware of simply pin a python version and a mypy version and leave it at that. Compatibility concerns for most of the upstreams just do not really ever consider that you'd be running *and* testing against a large spread of versions. I've gone the extra mile and I run mypy and pylint under all versions of python from 3.6 to 3.11 to ensure that developer workstations can run the same linting tests and feel assured that if it passes locally, it will pass on the CI and vice-versa. Our matrix is generally quite a bit larger than most upstreams, so I am playing a lot of whack-a-mole to keep things functioning consistently across the versions. I will probably even add support for Python 3.12 alpha *soon* because it's already available in the Fedora repo, and it will be good to find any compatibility issues before that version is officially released. (I'll need to do this for the qemu.qmp package, which should have support for 3.12 on the day 3.12 releases and not sometime afterwards.) I know the "check-tox" test was annoying upstream as it sometimes turned yellow because a new pylint version was released, but that's how I stay ahead of breaking changes before they're on local workstations. We could avoid at least one of the reasons for dropping 3.6 support by saying "Tough cookies, you'll use precisely this version of mypy and precisely this python interpreter, or you'll get nothing" and that does relieve a huge amount of pressure as-is. But, as a courtesy, I do go out of my way -- where possible -- to ensure that developers can use whichever versions of tools their distro is providing them so that they don't have to mess around with that stuff. Unfortunately, that means when upstreams start dropping support for older things, it gets hairier and quite a bit more painful. I think 3.6 being the first version that offers full-throated type hint support has unique pain in this circumstance because a lot of the details did not get hammered out until 3.7 or later; in general the amount of workarounds I have had to apply because something type checks only in 3.7+ but not 3.6 is fairly high compared to the number of times I've found that a different version was the odd one out. I don't expect this to become a recurring desire where I start to whine about old Python versions before our support window would otherwise let me drop them; I think this is actually just a unique pain point of this one version if we keep static typing. That's the tooling issue, anyway. > > Kevin >
John Snow <jsnow@redhat.com> writes: > On Tue, Feb 14, 2023 at 12:26 PM Kevin Wolf <kwolf@redhat.com> wrote: >> >> Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: >> > In the case of Python the issue is not the interpreter per se, though >> > there are a couple new feature in Python 3.7 that are quite nice (for >> > example improved data classes[1] or context variables[2]). The main >> > problem as far as I understood (and have seen in my experience) is >> > linting tools. New versions fix bugs that caused false positives, but >> > also become more strict at the same time. The newer versions at the >> > same time are very quick at dropping support for old versions of >> > Python; while older versions sometimes throw deprecation warnings on >> > new versions of Python. This makes it very hard to support a single >> > version of, say, mypy that works on all versions from RHEL8 and SLE15 >> > to Fedora 38 and Ubuntu 23.04. >> >> Why do we have to support a single version of mypy? What is wrong with >> running an old mypy version with old Python version, and a newer mypy >> with newer Python versions? > > Well, the problem is, ... > >> >> Sure, they will complain about different things, but it doesn't feel >> that different from supporting multiple C compilers in various versions. Different C compilers often produce different, useful warnings, so compiling with several of them can get you more useful warnings. However, you get pretty much all the useful warnings just from the latest versions, at least in my experience. You keep the old ones working not for getting additional useful warnings, but to compile on additional hosts. As the version range widens, warnings from old versions might even become such an annoyance that switching them off becomes sensible. > ...well, it's this. > > The first dimension of the test matrix is the version of mypy itself. > The second dimension of the test matrix is the version of Python mypy > runs under. A given version of mypy can run under multiple versions of > Python and may indeed have different behaviors under each. > The third dimension of this test matrix is the version(s) of Python > our code is targeting; for instance we configure mypy to understand > that we require Python 3.6. > > Trying to cast the net wide on *all of these* gets tough; the very > latest versions of mypy don't support 3.6 at all, so you'll have cases > where the mypy that just so happens to come with your Fedora > installation just won't work properly with our code anymore, which has > to test under 3.6 appropriately. > > In general, the majority of python package upstreams I am aware of > simply pin a python version and a mypy version and leave it at that. When pretty much everybody else doesn't do something we're doing, there are two obvious explanations: (1) we're so smart we discovered something worth doing pretty much first, or (2) the thing we're doing is a bad idea in general. In this case, my money is on (2). It may still be a good idea for us (because "in general"). But that notion needs to be supported with concrete reasons, such as ... > Compatibility concerns for most of the upstreams just do not really > ever consider that you'd be running *and* testing against a large > spread of versions. I've gone the extra mile and I run mypy and pylint > under all versions of python from 3.6 to 3.11 to ensure that developer > workstations can run the same linting tests and feel assured that if > it passes locally, it will pass on the CI and vice-versa. Our matrix > is generally quite a bit larger than most upstreams, so I am playing a > lot of whack-a-mole to keep things functioning consistently across the > versions. I will probably even add support for Python 3.12 alpha > *soon* because it's already available in the Fedora repo, and it will > be good to find any compatibility issues before that version is > officially released. (I'll need to do this for the qemu.qmp package, > which should have support for 3.12 on the day 3.12 releases and not > sometime afterwards.) > > I know the "check-tox" test was annoying upstream as it sometimes > turned yellow because a new pylint version was released, but that's > how I stay ahead of breaking changes before they're on local > workstations. > > We could avoid at least one of the reasons for dropping 3.6 support by > saying "Tough cookies, you'll use precisely this version of mypy and > precisely this python interpreter, or you'll get nothing" and that > does relieve a huge amount of pressure as-is. ... this one: > But, as a courtesy, I do > go out of my way -- where possible -- to ensure that developers can > use whichever versions of tools their distro is providing them so that > they don't have to mess around with that stuff. Unfortunately, that > means when upstreams start dropping support for older things, it gets > hairier and quite a bit more painful. I appreciate the courtesy. But it feels like it's gotten rather costly, and getting costlier. > I think 3.6 being the first version that offers full-throated type > hint support has unique pain in this circumstance because a lot of the > details did not get hammered out until 3.7 or later; in general the > amount of workarounds I have had to apply because something type > checks only in 3.7+ but not 3.6 is fairly high compared to the number > of times I've found that a different version was the odd one out. I > don't expect this to become a recurring desire where I start to whine > about old Python versions before our support window would otherwise > let me drop them; I think this is actually just a unique pain point of > this one version if we keep static typing. > > That's the tooling issue, anyway. Perhaps dropping just 3.6 from the test matrix will get us back to a reasonable cost / benefit ratio. But I'd like to encourage you to have a sober look at the test matrix, and not shy away from cutting more.
Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto: > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: > > In the case of Python the issue is not the interpreter per se, though > > there are a couple new feature in Python 3.7 that are quite nice (for > > example improved data classes[1] or context variables[2]). The main > > problem as far as I understood (and have seen in my experience) is > > linting tools. New versions fix bugs that caused false positives, but > > also become more strict at the same time. The newer versions at the > > same time are very quick at dropping support for old versions of > > Python; while older versions sometimes throw deprecation warnings on > > new versions of Python. This makes it very hard to support a single > > version of, say, mypy that works on all versions from RHEL8 and SLE15 > > to Fedora 38 and Ubuntu 23.04. > > Why do we have to support a single version of mypy? What is wrong with > running an old mypy version with old Python version, and a newer mypy > with newer Python versions? > > Sure, they will complain about different things, but it doesn't feel > that different from supporting multiple C compilers in various versions. > It's more like having to support only C++03 on RHEL 8 and only C++20 in Fedora 37, without even being able to use a preprocessor. For example old versions might not understand some type annotations and will fail mypy altogether, therefore even with newer versions you can't annotate the whole source and have to fall back to non-strict mode. Paolo > Kevin > >
On Tue, Feb 14, 2023 at 09:52:44PM +0100, Paolo Bonzini wrote: > Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto: > > > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: > > > In the case of Python the issue is not the interpreter per se, though > > > there are a couple new feature in Python 3.7 that are quite nice (for > > > example improved data classes[1] or context variables[2]). The main > > > problem as far as I understood (and have seen in my experience) is > > > linting tools. New versions fix bugs that caused false positives, but > > > also become more strict at the same time. The newer versions at the > > > same time are very quick at dropping support for old versions of > > > Python; while older versions sometimes throw deprecation warnings on > > > new versions of Python. This makes it very hard to support a single > > > version of, say, mypy that works on all versions from RHEL8 and SLE15 > > > to Fedora 38 and Ubuntu 23.04. > > > > Why do we have to support a single version of mypy? What is wrong with > > running an old mypy version with old Python version, and a newer mypy > > with newer Python versions? > > > > Sure, they will complain about different things, but it doesn't feel > > that different from supporting multiple C compilers in various versions. > > > > It's more like having to support only C++03 on RHEL 8 and only C++20 in > Fedora 37, without even being able to use a preprocessor. > > For example old versions might not understand some type annotations and > will fail mypy altogether, therefore even with newer versions you can't > annotate the whole source and have to fall back to non-strict mode. In terms of back compatibility, is there a distinction to be made between mypy compat and the python runtime compat ? If we add annotations wanted by new mypy, and old mypy doesn't understand them, that's not a huge problem. We can simply declare that we don't support old mypy, and skip the validation tests if old mypy is installed. The mypy results are targetted at upstream maintainers primarily, not people consuming QEMU, unless they are backporting huge amounts of code and need to validate it. IOW it should be sufficient to test once with an arbitrary version of mypy of our choosing. If we add annotations wanted by new mypy, and old python runtime barfs, then that's a significant problem, which would require us to either bump the min python or avoid the new mypy annotations. The same could be asked for the other linting tools we use like pylint / flake8. Is it sufficient to declare a min versions for those tools and skip the tests if not satisfied, while still retaining ability to execute the code on 3.6 ? Or are there some core python runtime features we also want to take advantage of at the same time ? With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé <berrange@redhat.com> writes: > On Tue, Feb 14, 2023 at 09:52:44PM +0100, Paolo Bonzini wrote: >> Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto: >> >> > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: >> > > In the case of Python the issue is not the interpreter per se, though >> > > there are a couple new feature in Python 3.7 that are quite nice (for >> > > example improved data classes[1] or context variables[2]). The main >> > > problem as far as I understood (and have seen in my experience) is >> > > linting tools. New versions fix bugs that caused false positives, but >> > > also become more strict at the same time. The newer versions at the >> > > same time are very quick at dropping support for old versions of >> > > Python; while older versions sometimes throw deprecation warnings on >> > > new versions of Python. This makes it very hard to support a single >> > > version of, say, mypy that works on all versions from RHEL8 and SLE15 >> > > to Fedora 38 and Ubuntu 23.04. >> > >> > Why do we have to support a single version of mypy? What is wrong with >> > running an old mypy version with old Python version, and a newer mypy >> > with newer Python versions? >> > >> > Sure, they will complain about different things, but it doesn't feel >> > that different from supporting multiple C compilers in various versions. >> > >> >> It's more like having to support only C++03 on RHEL 8 and only C++20 in >> Fedora 37, without even being able to use a preprocessor. >> >> For example old versions might not understand some type annotations and >> will fail mypy altogether, therefore even with newer versions you can't >> annotate the whole source and have to fall back to non-strict mode. > > In terms of back compatibility, is there a distinction to be > made between mypy compat and the python runtime compat ? > > If we add annotations wanted by new mypy, and old mypy doesn't > understand them, that's not a huge problem. We can simply declare > that we don't support old mypy, and skip the validation tests if > old mypy is installed. In theory, type hints are transparent at run time. In practice, use of modern type hints is known to break 3.6 at run time. I don't remember the details offhand, but John should be able to dig them up if you're interested. So, it should not be a problem, but it is. Checking types only with sufficiently new mypy still makes sense regardless. > The mypy results are targetted at upstream > maintainers primarily, not people consuming QEMU, unless they are > backporting huge amounts of code and need to validate it. IOW it > should be sufficient to test once with an arbitrary version of > mypy of our choosing. > > If we add annotations wanted by new mypy, and old python runtime > barfs, then that's a significant problem, which would require us > to either bump the min python or avoid the new mypy annotations. Yup. Avoiding new mypy leaves us with underpowered type checking. If dropping 3.6 had serious drawbacks, then we'd have to weigh type checking improvements against them. However, the only drawback we can see is "CentOS 8 provides no suitable Sphinx", which we can either accept or mitigate, I think. > The same could be asked for the other linting tools we use like > pylint / flake8. Is it sufficient to declare a min versions for > those tools and skip the tests if not satisfied, while still > retaining ability to execute the code on 3.6 ? Yes, let's try to narrow the version range for linting tools. > Or are there some core python runtime features we also want to > take advantage of at the same time ? Those are gravy.
On Thu, Feb 16, 2023 at 02:46:18AM +0100, Markus Armbruster wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > > On Tue, Feb 14, 2023 at 09:52:44PM +0100, Paolo Bonzini wrote: > >> Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto: > >> > >> > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: > >> > > In the case of Python the issue is not the interpreter per se, though > >> > > there are a couple new feature in Python 3.7 that are quite nice (for > >> > > example improved data classes[1] or context variables[2]). The main > >> > > problem as far as I understood (and have seen in my experience) is > >> > > linting tools. New versions fix bugs that caused false positives, but > >> > > also become more strict at the same time. The newer versions at the > >> > > same time are very quick at dropping support for old versions of > >> > > Python; while older versions sometimes throw deprecation warnings on > >> > > new versions of Python. This makes it very hard to support a single > >> > > version of, say, mypy that works on all versions from RHEL8 and SLE15 > >> > > to Fedora 38 and Ubuntu 23.04. > >> > > >> > Why do we have to support a single version of mypy? What is wrong with > >> > running an old mypy version with old Python version, and a newer mypy > >> > with newer Python versions? > >> > > >> > Sure, they will complain about different things, but it doesn't feel > >> > that different from supporting multiple C compilers in various versions. > >> > > >> > >> It's more like having to support only C++03 on RHEL 8 and only C++20 in > >> Fedora 37, without even being able to use a preprocessor. > >> > >> For example old versions might not understand some type annotations and > >> will fail mypy altogether, therefore even with newer versions you can't > >> annotate the whole source and have to fall back to non-strict mode. > > > > In terms of back compatibility, is there a distinction to be > > made between mypy compat and the python runtime compat ? > > > > If we add annotations wanted by new mypy, and old mypy doesn't > > understand them, that's not a huge problem. We can simply declare > > that we don't support old mypy, and skip the validation tests if > > old mypy is installed. > > In theory, type hints are transparent at run time. In practice, use of > modern type hints is known to break 3.6 at run time. I don't remember > the details offhand, but John should be able to dig them up if you're > interested. > > So, it should not be a problem, but it is. Ok, this is a pretty compelling motivating factor for dropping 3.6, as it is clear demonstration that we're being held back by the unfortunate lack of runtime compatibility. For most of the other problems we can simply mandate a new enough version of the add on tool, but if using new mypy requires annotations that break at runtime on 3.6 that's a painful blocker. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Am 14.02.2023 um 21:52 hat Paolo Bonzini geschrieben: > Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto: > > > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben: > > > In the case of Python the issue is not the interpreter per se, though > > > there are a couple new feature in Python 3.7 that are quite nice (for > > > example improved data classes[1] or context variables[2]). The main > > > problem as far as I understood (and have seen in my experience) is > > > linting tools. New versions fix bugs that caused false positives, but > > > also become more strict at the same time. The newer versions at the > > > same time are very quick at dropping support for old versions of > > > Python; while older versions sometimes throw deprecation warnings on > > > new versions of Python. This makes it very hard to support a single > > > version of, say, mypy that works on all versions from RHEL8 and SLE15 > > > to Fedora 38 and Ubuntu 23.04. > > > > Why do we have to support a single version of mypy? What is wrong with > > running an old mypy version with old Python version, and a newer mypy > > with newer Python versions? > > > > Sure, they will complain about different things, but it doesn't feel > > that different from supporting multiple C compilers in various versions. > > > > It's more like having to support only C++03 on RHEL 8 and only C++20 in > Fedora 37, without even being able to use a preprocessor. > > For example old versions might not understand some type annotations and > will fail mypy altogether, therefore even with newer versions you can't > annotate the whole source and have to fall back to non-strict mode. Sure, using old versions restricts which features can be used. But I suppose C++03 is (for the most part) a subset of C++20, and annotations understood by old mypy are a subset of annotation understood by new mypy? Which would make it something for the "shiny new feature" department. Don't get me wrong, I was among the first people advocating for using mypy and I still think any improvement in type checking is something we should take when we can. But while it may a very desirable new feature, it still is one. So I assume the real problem sits somewhere else? Or do we get lots of places where new mypy requires something that old mypy complains about and vice versa? Kevin
On Tue, Feb 14, 2023 at 03:03:54PM +0100, Paolo Bonzini wrote: > On Tue, Feb 14, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > [quote] > > The motivation for this series is that Python 3.6 was EOL at the end of > > 2021; upstream tools are beginning to drop support for it, including > > setuptools, pylint, mypy, etc. As time goes by, it becomes more > > difficult to support and test against the full range of Python versions > > that QEMU supports. The closer we get to Python 3.12, the harder it will > > be to cover that full spread of versions. > > [/quote] > > > > this is all about new/eol versions of software upstream, and I don't > > think that's a justification. QEMU explicitly aims to use distro provided > > versions and upstream EOL status is not relevant in that context. Even > > if using "pip" to install it is possible to limit yourself to upstream > > releases which still support 3.6. > > > > There is the separate issue of Meson dropping python 3.6 which motivates > > Paolo's series. Again though, we don't have to increase our minimum meson > > version, because meson is working today. It is our choice to to increase > > it to use latest available meson features. At some point we can decide > > what we have is good enough and we don't have to keep chasing the latest > > features. Maybe we're not there yet, but we should think about when that > > would be. > > In the case of Meson, the main advantage is moving _all_ of the > emulator configury out of the configure script. This requires > add_global_dependencies which was added in 0.63. So in that case it > is indeed mostly about shiny new features and it's not absolutely > necessary. > > In the case of Python the issue is not the interpreter per se, though > there are a couple new feature in Python 3.7 that are quite nice (for > example improved data classes[1] or context variables[2]). The main > problem as far as I understood (and have seen in my experience) is > linting tools. New versions fix bugs that caused false positives, but > also become more strict at the same time. The newer versions at the > same time are very quick at dropping support for old versions of > Python; while older versions sometimes throw deprecation warnings on > new versions of Python. This makes it very hard to support a single > version of, say, mypy that works on all versions from RHEL8 and SLE15 > to Fedora 38 and Ubuntu 23.04. > > [1] https://peps.python.org/pep-0557/ > [2] https://peps.python.org/pep-0567/ > > In fact this issue is the reason why RHEL9 does not package any of > these tools and does not run them as part of building RPMs even though > in principle it would be a good idea; it's too much of a pain to have > a single version that works across all the packages in the > distribution. > > Regarding your other suggestion: > > > * For non-native library/applications dependancies we aim > > to support only the most recent distro version. Users > > of older distros may need to dynamically fetch newer > > deps. > > I think this is a good idea, but one issue with "only supporting the > most recent distro version" is SUSE. While the most recent version of > SLE is about 5 years old, there is no next version in sight---SUSE > instead is working on their "Adaptable Linux Platform", but it's still > in the prototype stage[3]. So alternatively we could put a 4 or 5 year > cutoff after which you need to fetch newer deps. Considering the > delays between freeze and release of distros like RHEL or SLE, in > practice we would probably keep Python versions supported for 6-7 > years. Yeah, that kind of problem with very old SUSE would push towards simply excluding the LTS distros, or excluding them if they're older than N years, and expect users of such old distros to download newer python modules, etc. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Am 14.02.2023 um 08:40 hat Markus Armbruster geschrieben: > I read this on Friday, and decided to let it sit until after the > weekend. Well, it's now Tuesday, and to be frank, it's still as > offensively flippant as it was on Friday. It shows either ignorance of > or cavalier disregard for the sheer amount of work some of us have had > to put into keeping old versions of Python viable. > > The latter would be quite unlike you, so it must be the former. Honest question, Markus, because I haven't been following as much what is happening in Python recently: What are the biggest pain points in this context? Has Python started removing features from new versions more aggressively so that we have to update the code so it can run on newer versions, and still keep compatibility paths for older versions that don't have the replacement yet? Kevin
Kevin Wolf <kwolf@redhat.com> writes: > Am 14.02.2023 um 08:40 hat Markus Armbruster geschrieben: >> I read this on Friday, and decided to let it sit until after the >> weekend. Well, it's now Tuesday, and to be frank, it's still as >> offensively flippant as it was on Friday. It shows either ignorance of >> or cavalier disregard for the sheer amount of work some of us have had >> to put into keeping old versions of Python viable. >> >> The latter would be quite unlike you, so it must be the former. > > Honest question, Markus, because I haven't been following as much what > is happening in Python recently: What are the biggest pain points in > this context? > > Has Python started removing features from new versions more aggressively > so that we have to update the code so it can run on newer versions, and > still keep compatibility paths for older versions that don't have the > replacement yet? A caveat before I answer: I'm aware of four non-trivial uses of Python, (QAPI generator, Avocado, the qemu/qmp package, meson), but my direct experience is limited to just QAPI. And another preliminary remark: I think there's confusion about the actual impact of dropping 3.6. But attempting to clarify that is a topic for another message (Message-ID: <87v8k2ycjb.fsf@pond.sub.org>, just sent). I've personally observed roughly three kinds of pain. In order of (sharply) increasing intensity: 1. Python language and core libraries Python still evolves at a good pace. We can sidestep new features. We can't sidestep bugs. In either case, we need to become aware of the issue first. For features, diligent reading of documentation suffices, but for bugs, it's testing or bust. Things have exploded into John's face often enough to "motivate" him to spend a huge chunk of his time (man-months) on building and maintaining infrastructure for testing with all the Python versions. Naturally, the wider the version range, the bigger the test matrix, and the deeper the time sink. The range of versions we're trying to target is actually wider than Python's deprecation grace period. This means we sometimes get to do things in both the old and the new way (because neither works across the whole range of versions), or look for ways to sidestep the mess somehow. Python programs are nicely portable between host systems. Between more than a few Python versions, not so much. 2. Python tooling and non-core libraries Much of these don't give a rat's ass for anything but the latest few versions, let alone for EOLed ones like 3.6. Likely because they decided it would be a fool's errand. For instance, Python packaging has changed beyond recognition since 3.6. There is no single way to package that spans the range of versions we try to target. Bites the qemu/qmp package. Linters are also a moving target, and keeping lint tests pass for a wide range of linters version involves lots of largely pointless fiddling. Another dimension in the test matrix. 3. Python typing is too immature in anything but latest versions This is kind of a special case of 2., but it's a big one for *me*, so I'm making it an item of its own. Type hints have been evolving substantially, and it doesn't look like this is going to stop any time soon. Trying to keep typing tests pass for a wide range of mypy versions is even less practical than for other linters. Can we simply skip type checking unless we have a sufficiently recent mypy? After all, type hints are designed to be transparent at runtime, so code with new type hints should still run on old Python, shouldn't it? Nope. Doesn't. Kevin and others pointed out that the QAPI generator code is harder to work with than it should be, in good part because they have to spend too much time understanding possible types. In response to this valid complaint, we embarked on the known non-trivial project to add type hints to the QAPI code generator. This may have been a mistake. We had no idea just how much trouble type hints would give us when combined with a range of Python versions this wide. But here we are, 2+ years older and somewhat better informed, and our informed desire is to narrow the version range as much as practical. Ditching 3.6 feels eminently practical. Let's do it.
On Tue, 14 Feb 2023 at 07:40, Markus Armbruster <armbru@redhat.com> wrote: > I read this on Friday, and decided to let it sit until after the > weekend. Well, it's now Tuesday, and to be frank, it's still as > offensively flippant as it was on Friday. It shows either ignorance of > or cavalier disregard for the sheer amount of work some of us have had > to put into keeping old versions of Python viable. From my point of view it is definitely ignorance. I simply didn't expect that a serious major scripting language would be making it this hard to write code that is backwards compatible with older but not ancient versions. thanks -- PMM
On 14/02/2023 08.40, Markus Armbruster wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > [...] > >> We don't have to drop python 3.6. It is a choice because >> of a desire to be able to use some shiny new python >> features without caring about back compat. > > I read this on Friday, and decided to let it sit until after the > weekend. Well, it's now Tuesday, and to be frank, it's still as > offensively flippant as it was on Friday. It shows either ignorance of > or cavalier disregard for the sheer amount of work some of us have had > to put into keeping old versions of Python viable. I'm a complete python ignorant, too, so I'm a little bit surprised of the amount of pain that these scripts are causing. No matter of that fact, I think Peter still has a point that we have a real conflict here with our current support policy. So this either means that Python was the wrong choice for our needs (since it is moving too fast and causing too much friction), or we should really rethink our support policy. I guess we're too deep into the Python rabbit hole already, and I'm not aware of any other good solutions (back to Perl scripts? No, thanks!), so it's likely quite impossible to tune that knob. Thus we should maybe really start talking about our support policy now. I think the main problem is likely the sentence "Support for the previous major version will be dropped 2 years after the new major version is released". Maybe we should shorten that time frame to 1 year. The 2 years caused some confusions in the past already, since e.g. Debian only supports the previous major release for only one more year, and macOS also releases a major version each year ... so IMHO we could shorten the time frame for the previous major release to 1 year instead. People then could still continue building QEMU on CentOS 8, but they have to be aware that they might install other software like Sphinx manually if they want to continue using QEMU with docs there. What do you think? Thomas
On Tue, Feb 14, 2023 at 09:35:44AM +0100, Thomas Huth wrote: > On 14/02/2023 08.40, Markus Armbruster wrote: > > Daniel P. Berrangé <berrange@redhat.com> writes: > > > > [...] > > > > > We don't have to drop python 3.6. It is a choice because > > > of a desire to be able to use some shiny new python > > > features without caring about back compat. > > > > I read this on Friday, and decided to let it sit until after the > > weekend. Well, it's now Tuesday, and to be frank, it's still as > > offensively flippant as it was on Friday. It shows either ignorance of > > or cavalier disregard for the sheer amount of work some of us have had > > to put into keeping old versions of Python viable. > > I'm a complete python ignorant, too, so I'm a little bit surprised of the > amount of pain that these scripts are causing. > > No matter of that fact, I think Peter still has a point that we have a real > conflict here with our current support policy. So this either means that > Python was the wrong choice for our needs (since it is moving too fast and > causing too much friction), or we should really rethink our support policy. > > I guess we're too deep into the Python rabbit hole already, and I'm not > aware of any other good solutions (back to Perl scripts? No, thanks!), so > it's likely quite impossible to tune that knob. I still believe python is a probably the best thing for what we're using it for. Certainly would not suggest shell or perl, and using a compiled language would add its own complications for cross compilation. > Thus we should maybe really start talking about our support policy now. I > think the main problem is likely the sentence "Support for the previous > major version will be dropped 2 years after the new major version is > released". Maybe we should shorten that time frame to 1 year. The 2 years > caused some confusions in the past already, since e.g. Debian only supports > the previous major release for only one more year, and macOS also releases a > major version each year ... so IMHO we could shorten the time frame for the > previous major release to 1 year instead. People then could still continue > building QEMU on CentOS 8, but they have to be aware that they might install > other software like Sphinx manually if they want to continue using QEMU with > docs there. What do you think? I think perhaps the problem is not in the length of time defined by our support policy, but rather that we're facing a rather different reality to the one we've historically been used it, where distros are no longer critical dependancies and our support policy does not reflect that. For any C/C++ application, wanting to target the versions shipped in a distro has been pretty much normal practice. C has not ever come with a standard package manager toolset, the distros service that role. The distros also aren't generally a fan of shipping multiple versions of C libs in parallel. Pretty much every non-C library though is different. They all have their own package manager service / tools (perl has cpan, pytyhon has PyPi/pip, ruby has gems. With latest compiled languages like Go/Rust, this has gone one step further and is natively integrated into the compiler toolchain as standard. IOW, for everything except C, it has become increasingly normal practice to ignore the distro and dynamically download all the deps your application needs into a self contained local environment. Now, the distros aren't especially a fan of this new world, since they still prefer to unbundle all these deps, but I think that approach is increasingly difficult for them to achieve because the majority of upstreams don't care for the distro versions. Thus what we're experiancing is a clash between the traditional way that C applications/libraries deal with their deps, vs the way pretty much every other language deals with their deps in the modern world. It has come up now because we're making much more use of python now, than we did in the past. Our support policy is written from the POV of the C world, and merely reducing the length of time we support a distro does not address the different world view of Python. Should we instead try to be more explicit about the different needs of the non-C dependencies ? We could for example say * For native library/application dependancies we aim to support the two most recent distro versions, for 2 years overlap * For non-native library/applications dependancies we aim to support only the most recent distro version. Users of older distros may need to dynamically fetch newer deps. The python 3.8 runtime would be considered a native dep, so fall under the 2 distro versions rule. This is fine with CentOS 8, since it provides newer python runtime versions. The python libraries, or tools written in python (meson), would fall under the second rule, and so only need to target one distro version. This would be compatible with CentOS 8, as the users would be expected to download extra python components (or we do it on their behalf). For the second rule, rather than saying most recent distro versions, possibly we might want to carve out an exclusion for LTS distros too. ie, explicitly don't care about versions of non-native bits in RHEL at all, beyond availability of the base (python) runtime. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé <berrange@redhat.com> writes: > On Tue, Feb 14, 2023 at 09:35:44AM +0100, Thomas Huth wrote: >> On 14/02/2023 08.40, Markus Armbruster wrote: >> > Daniel P. Berrangé <berrange@redhat.com> writes: >> > >> > [...] >> > >> > > We don't have to drop python 3.6. It is a choice because >> > > of a desire to be able to use some shiny new python >> > > features without caring about back compat. >> > >> > I read this on Friday, and decided to let it sit until after the >> > weekend. Well, it's now Tuesday, and to be frank, it's still as >> > offensively flippant as it was on Friday. It shows either ignorance of >> > or cavalier disregard for the sheer amount of work some of us have had >> > to put into keeping old versions of Python viable. >> >> I'm a complete python ignorant, too, so I'm a little bit surprised of the >> amount of pain that these scripts are causing. >> >> No matter of that fact, I think Peter still has a point that we have a real >> conflict here with our current support policy. So this either means that >> Python was the wrong choice for our needs (since it is moving too fast and >> causing too much friction), or we should really rethink our support policy. >> >> I guess we're too deep into the Python rabbit hole already, and I'm not >> aware of any other good solutions (back to Perl scripts? No, thanks!), so >> it's likely quite impossible to tune that knob. > > I still believe python is a probably the best thing for what we're using > it for. Certainly would not suggest shell or perl, and using a compiled > language would add its own complications for cross compilation. > >> Thus we should maybe really start talking about our support policy now. I >> think the main problem is likely the sentence "Support for the previous >> major version will be dropped 2 years after the new major version is >> released". Maybe we should shorten that time frame to 1 year. The 2 years It's actually "2 years after the new major version is released or when the vendor itself drops support, whichever comes first." >> caused some confusions in the past already, since e.g. Debian only supports >> the previous major release for only one more year, and macOS also releases a >> major version each year ... so IMHO we could shorten the time frame for the >> previous major release to 1 year instead. People then could still continue >> building QEMU on CentOS 8, but they have to be aware that they might install >> other software like Sphinx manually if they want to continue using QEMU with >> docs there. What do you think? > > > I think perhaps the problem is not in the length of time defined by > our support policy, but rather that we're facing a rather different > reality to the one we've historically been used it, where distros > are no longer critical dependancies and our support policy does not > reflect that. > > > For any C/C++ application, wanting to target the versions shipped in a > distro has been pretty much normal practice. C has not ever come with > a standard package manager toolset, the distros service that role. The > distros also aren't generally a fan of shipping multiple versions of > C libs in parallel. > > > Pretty much every non-C library though is different. They all have > their own package manager service / tools (perl has cpan, pytyhon has > PyPi/pip, ruby has gems. With latest compiled languages like Go/Rust, > this has gone one step further and is natively integrated into the > compiler toolchain as standard. > > > IOW, for everything except C, it has become increasingly normal > practice to ignore the distro and dynamically download all the deps > your application needs into a self contained local environment. > Now, the distros aren't especially a fan of this new world, since > they still prefer to unbundle all these deps, but I think that > approach is increasingly difficult for them to achieve because the > majority of upstreams don't care for the distro versions. > > > Thus what we're experiancing is a clash between the traditional > way that C applications/libraries deal with their deps, vs the > way pretty much every other language deals with their deps in > the modern world. It has come up now because we're making much > more use of python now, than we did in the past. Yes. The traditional way of building applications is to examine the environment, and configure the application accordingly. If you depend on libfoo, configure looks for (a supported version of) libfoo, and the source code deals with differences between versions, if any. libfoo is expected to bend over backwards to avoid differences. The newfangled way of building applications is to set up a controlled environment. No need to configure the application. Pros and cons, no need to rehash them here, I believe. > Our support policy is written from the POV of the C world, and > merely reducing the length of time we support a distro does not > address the different world view of Python. > > Should we instead try to be more explicit about the different > needs of the non-C dependencies ? > > We could for example say > > * For native library/application dependancies we aim to > support the two most recent distro versions, for 2 years > overlap > > * For non-native library/applications dependancies we aim > to support only the most recent distro version. Users > of older distros may need to dynamically fetch newer > deps. Who does the fetching, users manually, or the build process automatically? > The python 3.8 runtime would be considered a native dep, so fall > under the 2 distro versions rule. This is fine with CentOS 8, > since it provides newer python runtime versions. > > The python libraries, or tools written in python (meson), would > fall under the second rule, and so only need to target one distro > version. This would be compatible with CentOS 8, as the users would > be expected to download extra python components (or we do it on > their behalf). > > For the second rule, rather than saying most recent distro versions, > possibly we might want to carve out an exclusion for LTS distros too. > ie, explicitly don't care about versions of non-native bits in RHEL > at all, beyond availability of the base (python) runtime. Interesting idea. Can anyone think of reasons not to do this?
On Thu, Feb 16, 2023 at 02:08:33AM +0100, Markus Armbruster wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > Our support policy is written from the POV of the C world, and > > merely reducing the length of time we support a distro does not > > address the different world view of Python. > > > > Should we instead try to be more explicit about the different > > needs of the non-C dependencies ? > > > > We could for example say > > > > * For native library/application dependancies we aim to > > support the two most recent distro versions, for 2 years > > overlap > > > > * For non-native library/applications dependancies we aim > > to support only the most recent distro version. Users > > of older distros may need to dynamically fetch newer > > deps. > > Who does the fetching, users manually, or the build process > automatically? I expect both cases need supporting. In the case of distro builds, they will have to fetch the deps ahead of time, because the build environments usually won't have any network access. Some contributors have also previously objected to the build system fetching stuff off the network, but they're a small minority. For friendliness to developers, the build process would be best to fetch automatically, if they haven't been provided upfront. > > The python 3.8 runtime would be considered a native dep, so fall > > under the 2 distro versions rule. This is fine with CentOS 8, > > since it provides newer python runtime versions. > > > > The python libraries, or tools written in python (meson), would > > fall under the second rule, and so only need to target one distro > > version. This would be compatible with CentOS 8, as the users would > > be expected to download extra python components (or we do it on > > their behalf). > > > > For the second rule, rather than saying most recent distro versions, > > possibly we might want to carve out an exclusion for LTS distros too. > > ie, explicitly don't care about versions of non-native bits in RHEL > > at all, beyond availability of the base (python) runtime. > > Interesting idea. Can anyone think of reasons not to do this? It is probably even more compelling when looking at SLES, which is having an even larger gap between releases than RHEL does. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thomas Huth <thuth@redhat.com> writes: > On 14/02/2023 08.40, Markus Armbruster wrote: >> Daniel P. Berrangé <berrange@redhat.com> writes: >> [...] >> >>> We don't have to drop python 3.6. It is a choice because >>> of a desire to be able to use some shiny new python >>> features without caring about back compat. >> I read this on Friday, and decided to let it sit until after the >> weekend. Well, it's now Tuesday, and to be frank, it's still as >> offensively flippant as it was on Friday. It shows either ignorance of >> or cavalier disregard for the sheer amount of work some of us have had >> to put into keeping old versions of Python viable. > > I'm a complete python ignorant, too, so I'm a little bit surprised of > the amount of pain that these scripts are causing. > > No matter of that fact, I think Peter still has a point that we have a > real conflict here with our current support policy. So this either > means that Python was the wrong choice for our needs (since it is > moving too fast and causing too much friction), or we should really > rethink our support policy. > > I guess we're too deep into the Python rabbit hole already, and I'm > not aware of any other good solutions (back to Perl scripts? No, > thanks!), so it's likely quite impossible to tune that knob. > > Thus we should maybe really start talking about our support policy > now. I think the main problem is likely the sentence "Support for the > previous major version will be dropped 2 years after the new major > version is released". Maybe we should shorten that time frame to 1 > year. I think this should be a fair approach. Generally I recommend avoiding installing a new LTS until at least the first tick release has ironed out the obvious bugs. A year seems like a fair grace period to update to the next LTS. Those that like sitting on old maintained LTS releases are less likely to be tracking the bleeding edge of development anyway and likely are happy with their distro provided packages. BTW my next testing/next finally updates the last few Ubuntu 20.04 to 22.04 systems which also allows removing a few tsan and clang hacks in the process. Progress might not be a straight line but it averages out in that approximate direction ;-) > The 2 years caused some confusions in the past already, since > e.g. Debian only supports the previous major release for only one more > year, and macOS also releases a major version each year ... so IMHO we > could shorten the time frame for the previous major release to 1 year > instead. People then could still continue building QEMU on CentOS 8, > but they have to be aware that they might install other software like > Sphinx manually if they want to continue using QEMU with docs there. > What do you think? Works for me at least. -- Alex Bennée Virtualisation Tech Lead @ Linaro
On Fri, 10 Feb 2023 at 16:52, Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Fri, Feb 10, 2023 at 04:32:19PM +0000, Peter Maydell wrote: > > On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote: > > > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote: > > >> This confuses me. We work fine with Python 3.6 today. > > > > > > > > > That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments. > > > > Something somewhere seems kind of out-of-sync here. Either: > > * Python is deprecating old versions too quickly and > > dropping support for them too fast > > Nope, they're fine in declaring EOL whenever they like. There's > no requirement for upstreams to support arbitrary old versions > for any length of time. To be clear, yes, absolutely any other software project can set whatever lifecycle it chooses to do, and similarly distros can do what they wish to do. But somewhere along the line something's gone sideways here, because all these choices people are making seem reasonable and yet here we are :-) I put this bit in here because one response to "upstream's idea of how long it wants to support something is very short" would be "this isn't a dependency that it's suitable for QEMU as a project to have, because the mismatch is too great". (This would obviously be super-painful in this specific case...) > > * CentOS is retaining old versions of Python when it needs to > > ship newer ones > > It is also totally OK for an distro to ship and support software > versions which are EOL upstream. In fact for enterprise distros > you can generally assume that *all* the software versions they > ship are probably EOL or nearly so. The main value of enterprise > distros is that they provide long term support, where upstreams > are not willing to. But we as a project also have the choice, if it seems to us that Distro X is shipping absolutely ancient versions of dependencies, to say "we don't support distro X, because its life cycle policies are too far out of sync with ours", or to say "you're going to need to provide the dependencies yourself, here's a suggestion". (We effectively do that already with macos and Windows.) > > * Our policy for what distros we support is overly lax > > and causing us to try to support ancient platforms > > that we shouldn't be trying to support > > I don't think so. Users of distros are not anywhere near > as aggressive at updating their installations as users > are. The number of users of RHEL-8 will dwarf that of > RHEL-9 by a large margin for a decent amount of time > yet. Right. All of these things together seem to say: * Python is not an unreasonable thing for the project to depend on * CentOS 8 is not an unreasonable distro for us to want to continue to support * Therefore we should continue to work with the Python that ships with CentOS 8... [snip] > We don't have to drop python 3.6. It is a choice because > of a desire to be able to use some shiny new python > features without caring about back compat. > > Similarly we don't have to use the new meson which drops > support for python 3.6, it is again a choice because we > want to rely on some new meson features. > > QEMU could easily carry on supporting python 3.6 until > the affected distros drop out ofo our support matrix, but > we would have to opt out of using certain new features, > or put more effort into back compat. > > Personally I'm still on the side of carrying on supporting > 3.6 per our distro support policy, but if broad consensus > is to drop 3.6 I'm not going push back anymore. This is really where I'm at as well -- we set our distro support policy, and we know that means that sometimes we have to deal with continuing to support older versions of dependencies even when it might be easier for us if we could require newer versions. I'm reluctant to say that Python gets a special derogation from that policy. thanks -- PMM
On 2/10/23 18:15, Peter Maydell wrote: > Right. All of these things together seem to say: > * Python is not an unreasonable thing for the project > to depend on > * CentOS 8 is not an unreasonable distro for us to > want to continue to support > * Therefore we should continue to work with the Python > that ships with CentOS 8... > > [snip] > >> We don't have to drop python 3.6. It is a choice because >> of a desire to be able to use some shiny new python >> features without caring about back compat. >> >> Similarly we don't have to use the new meson which drops >> support for python 3.6, it is again a choice because we >> want to rely on some new meson features. >> >> QEMU could easily carry on supporting python 3.6 until >> the affected distros drop out ofo our support matrix, but >> we would have to opt out of using certain new features, >> or put more effort into back compat. >> >> Personally I'm still on the side of carrying on supporting >> 3.6 per our distro support policy, but if broad consensus >> is to drop 3.6 I'm not going push back anymore. > > This is really where I'm at as well -- we set our distro > support policy, and we know that means that sometimes > we have to deal with continuing to support older versions > of dependencies even when it might be easier for us if we > could require newer versions. There are four possibilities: * we change the support policy and stop supporting CentOS 8 and SLE 15, not a good idea since a lot of people have not migrated to CentOS 9 yet. * we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being supported. This means several more years since SLE 16 hasn't even been released. * we support Python 3.6 just for building documentation, i.e. we are careful not to use Python 3.7+ features in our Sphinx extensions but are free to use them elsewhere. CentOS 8 users can install sphinx from either RPMs (which will use Python 3.6) or pip (which will use Python 3.8). * we only support Python 3.7+, which means CentOS 8 CI and users have to either install Sphinx from pip or disable documentation. The only difference between the last two is documentation and CI configuration. The code is exactly the same. > I'm reluctant to say that > Python gets a special derogation from that policy. Note that its not the Python runtime but the Python dependencies, for which we already install avocado and some Python development tools in a virtual environment. So, the questions are: * to what extent can we rely on pip packages (preinstalled by the user) instead of the distro packages? * to what extent should QEMU install its own dependencies via pip in a virtual environment instead of requiring the user to preinstall them? Right now the answers for both are "avocado gets an exception for tests/, Python development tools such as mypy get an exception for python/". The Python 3.7+ series (not this one by John) currently adds "sphinx gets an exception to the first answer only". In the future I would like to unify virtual environment generation for tests/ and python/ and move it to configure. If desirable, this would make it possible to implement something like "the user need not care about Python dependencies, configure can (but need not) install them all via pip". Distros would still package the dependencies, but users would have a slightly easier time building QEMU. Thanks, Paolo
On Fri, Feb 10, 2023 at 07:27:41PM +0100, Paolo Bonzini wrote: > On 2/10/23 18:15, Peter Maydell wrote: > There are four possibilities: > > * we change the support policy and stop supporting CentOS 8 and SLE 15, not > a good idea since a lot of people have not migrated to CentOS 9 yet. > > * we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being > supported. This means several more years since SLE 16 hasn't even been > released. The massive gap from SLE 15 to 16 is something our support policy wasn't anticipating, though it probably should have done, since a 4-5 year gap is actually normal if looking at previous SLE versions. I was thinking too much about RHEL which has a short 3 year gap, but also rebases software versions in .y releases, so often the dependancies aren't as old as the 3 year life of RHEL would suggest. We need to finese our support policy to give us more flexibility wrt the very long life / long gap enterprise distros. > * we support Python 3.6 just for building documentation, i.e. we are careful > not to use Python 3.7+ features in our Sphinx extensions but are free to use > them elsewhere. CentOS 8 users can install sphinx from either RPMs (which > will use Python 3.6) or pip (which will use Python 3.8). > > * we only support Python 3.7+, which means CentOS 8 CI and users have to > either install Sphinx from pip or disable documentation. > > The only difference between the last two is documentation and CI > configuration. The code is exactly the same. While it is good to mention the idea of 3.6 just for docs, I don't think it is really a good idea to put into practice. IMHO there's compelling benefit in having a clear & simple story for the min versions of dependancies, both for us as maintainers, and for people consuming it. So if we want 3.7, we should apply it universally without caveats / carve-outs. > > I'm reluctant to say that > > Python gets a special derogation from that policy. > > Note that its not the Python runtime but the Python dependencies, for which > we already install avocado and some Python development tools in a virtual > environment. So, the questions are: > > * to what extent can we rely on pip packages (preinstalled by the user) > instead of the distro packages? > > * to what extent should QEMU install its own dependencies via pip in a > virtual environment instead of requiring the user to preinstall them? FWIW, when I introduced scripts/git-submodule.sh script to handle automatically checking out git submodules during build, there was quite a bit of negative feedback from some users/contributors who don't want stuff being fetched off the net during builds. Typically their scenario was that they have a QEMU checkout on one machine with net access. That filesystem was exported to another machine (or multiple other machines) to perform the actual build(s) and those build machines lacked net access. This motiviated the introduction of the configure arg, to tell us *NOT* to attempt to checkout submodules, but merely validate that they exist at the right checkout hash. The user would manually checkout the submodules on the hosts which had net access. This situation is not too dis-similar to what distros have in their package build systems which often block net access. None of this means we can't use a private virtualenv for all of QEMU python bits. Just that if we go this route, then we're putting a greater burden on people whose distro doesn't have the required packages, and whose build env lacks net access. They'll have to populate a venv manually in some way and get it onto their build env. The number of people with this situation though likely is small enough, that it is likely an acceptable burden / tradeoff in general. The burden would only apply to the pieces of build that are actually mandatory for build, the most prominent of which are meson, and the QAPI generation code. Bits like avocado, mypy, flake8, etc are all optional extra tests. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 10/2/23 01:31, John Snow wrote: > CentOS 8 does not ship with a sphinx new enough for our purposes (It > necessarily uses Python 3.6), so drop this from this build. We can > resume building docs on CentOS 9 if we wish, but we also currently test > and build docs on Fedora, Ubuntu, Alpine and Debian. This wish ... > Signed-off-by: John Snow <jsnow@redhat.com> > --- > .gitlab-ci.d/buildtest.yml | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml > index 0aa149a3524..0eb7f6606d4 100644 > --- a/.gitlab-ci.d/buildtest.yml > +++ b/.gitlab-ci.d/buildtest.yml > @@ -166,7 +166,7 @@ build-system-centos: > variables: > IMAGE: centos8 ... can be remembered by adding a "Please add --enable-docs when upgrading to centos9" comment here. > CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system > - --enable-modules --enable-trace-backends=dtrace --enable-docs > + --enable-modules --enable-trace-backends=dtrace > --enable-vfio-user-server > TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu > x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
© 2016 - 2026 Red Hat, Inc.