[Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021. Reasons: 1. Python 3.5.9 will be the final release of Python 3.5, in November 2019 ( https://www.python.org/dev/peps/pep-0478/ ) 2. Python 3.6.0 was released in December 2016 ( https://www.python.org/dev/peps/pep-0494/ ) 3. PyPy 7.1.1 introduced Python 3.6 support in April 2019 ( https://morepypy.blogspot.com/2019/04/pypy-711-bug-fix-release.html ). PyPy 7.3.1 is the latest PyPy version ( https://morepypy.blogspot.com/2020/04/pypy-731-released.html ). 4. Python 3.6 introduced support for variable annotations ( https://www.python.org/dev/peps/pep-0526/ ) 5. Python 3.6 introduced new string interpolation (f-string) support ( https://www.python.org/dev/peps/pep-0498/ ) 6. Python 3.6 introduced support for asynchronous generators ( https://www.python.org/dev/peps/pep-0525/ ) and asynchronous comprehensions ( https://www.python.org/dev/peps/pep-0530/ ) Based on item 2., Python 3.6.0 has been around for almost 4 years, which is a reasonable amount of time. Based on item 3., PyPy supports Python 3.6. It might be nice to take advantage of some of the language features introduces in items 4-6. I think dropping support one year from now in 2021, would give sufficient notice, and be in line with the Twisted Compatibility Policy ( https://twistedmatrix.com/documents/current/core/development/policy/compatib... ). None of these items are critial, but I thought I would ask, to get people's opinions. -- Craig
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix). Why I think this is probably fine: - Debian Stable (buster) has 3.7. oldstable (stretch) has 3.5.3, but I don't think we care about oldstable. - Ubuntu 18.04 has Python 3.6. 16.04 has 3.5.1, which is a sucky, buggy version that people shouldn't use anyway. Plus, getting a modern Python on Ubuntu isn't shockingly hard (deadsnakes makes it trivial). - CentOS and RHEL 6 and 7 have Software Collections which makes Python 3.6+ readily available. - Windows has 3.8 available in the Store. - MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily. - For everyone else, there's always Docker. So, I guess if we have 3.8 support, the next release can announce the dropping of 3.5, and then the release after that can be the final one with support. - Amber On 14/5/20 5:10 am, Craig Rodrigues wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) <hawkowl@atleastfornow.net> wrote:
I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5. The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy. -g
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality (due to it being provisional then?) which means things like Eliot doesn't work due to aiocontextvars barfing on missing functionality. There's also 20.04 LTS now, making 16.04 old-old-LTS. I don't think it's worth it. - Amber On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote:
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
Personally, I don't have any dependencies on Python 3.5. I just suggested once year's notice to give people time to adjust, but if no one really cares, dropping 3.5 support some time this year is fine by me. Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support? -- Craig On Wed, May 13, 2020 at 10:37 PM Amber Brown <hawkowl@atleastfornow.net> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.
Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I don't want to be putting (non-trivial numbers of) users into the position of having to skip a bunch of updates. -glyph
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:
Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that work completed by the end of the year. In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure. ---Tom
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Glad to hear it :).
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure.
Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :). -g
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure. Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :).
Oh it's not for lack of wanting. :-) The service in question is actually spawning Python processes, so posix_spawn support doesn't seem likely to help much: fork() + exec() is a drop in the bucket compared to Python imports, particularly in Python 3 (I'm hoping to explore oxidized-importer [1] to help with this at some point). If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so. ---Tom [1]: https://pypi.org/project/oxidized-importer/
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
Would putting all the .pyc/.pyo into a zip file help with the import problem? You only have one file to open, the .zip, and no compile as its only .pyc/.pyo.
Why not waitpid for all children (pid=0 or pid=-1) in a single call? Barry
---Tom
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely? Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!) Anyway there's lot of exciting room for performance improvements here, it would be nice to have some benchmarks for this kind of high-load process-monitor application. -g
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely?
Those all sound like reasonable optimizations with little downside to me.
Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!)
Nonono, that was a synthetic stress test. It did spawn Python processes, but they immediately called `exec sleep` to conserve memory. The result of the test was that the service would run out of memory _well_ before inefficiencies in Twisted's reaping scheme became a problem. ---Tom
![](https://secure.gravatar.com/avatar/e5a514e14e44913930aa1ac15f508746.jpg?s=120&d=mm&r=g)
On 16/05/2020 06:56, Glyph wrote:
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7... I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions? It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy. Or, for that matter, why not just run in a Docker container? Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it? -glyph
![](https://secure.gravatar.com/avatar/89b7125ce22cf40fd72ffebbc54ea8b4.jpg?s=120&d=mm&r=g)
On 23/05/2020 06:39, Glyph wrote:
I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever). Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure. Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode), it might become easier to justify that it really /is/ reasonable for us to require a newer version of Python (and that they should really really upgrade their debian version, and its their own fault that they haven't and have to deal with faff that comes from that). This would also be easier if debian had newer versions of python in backports, or someone ran a semi-official package repository which had them, but as far as I can tell no one does for debian. (Of course you can argue that all the above is a bit silly from a security perspective, especially when you start considering virtualenvs and the like, but I have some sympathy for their outlook even if it is a pain at times). In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment? Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner. (FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs) - Erik
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I think this sounds like a misunderstanding of Debian's vetting process? It's not like there's a ton of additional auditing that goes into packaging something. There's definitely an authentication process for both Twisted and Python, although this attestation could be somewhat stronger and less centralized, PyPI does quite a bit of heavy lifting there.
Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure.
It's worth noting that Python 3.5 itself is out of security support (per https://www.python.org/dev/peps/pep-0478/ <https://www.python.org/dev/peps/pep-0478/>) and it's not like Debian employs a team of Python security-support engineers to continue maintaining this dead branch. If their infrastructure were to try to comply with various security standards such as PCI-DSS, you might bump into the need to upgrade to a newer version with available future patch releases anyway.
Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode),
You say "Autumn", but... oldstable is supported for the purposes of migrating off of it... it's not really even supported by Debian proper. To quote from https://wiki.debian.org/DebianOldStable <https://wiki.debian.org/DebianOldStable> :
The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.
According to https://wiki.debian.org/DebianBuster <https://wiki.debian.org/DebianBuster>, Buster was released on 2019-07-06, which means we've now got less than one month of even this small amount of support that is offered; i.e. the expectation from Debian's perspective is that they'd be done migrating off of oldstable by that point. I guess https://wiki.debian.org/LTS <https://wiki.debian.org/LTS> is a separate project?
It's a bit of a pet peeve of mine for reasons which are mostly covered by this blog post -https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/ <https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/> But it bugs me not just because of the expectation of free labor (we can refuse to support 3.5 if that is a point of contention, as you say) but because "security support" is a bit of a misnomer, especially in Debian's case where there's no commercial relationship and hence not much in the way of libability. Much to my chagrin I can't find the paper I remember reading about this right now, but, I believe these things are all true: Many non-"security" bugs are in fact security bugs that nobody has noticed you can exploit. Many "low-severity" or "un-exploiable" security bugs are in fact exploitable "supported" distros rarely take care to backport many patches for their software, and when they do, they often make undetected errors (like debian's infamous ssh bug) which are analyzed by far fewer security analysts than the upstream source code. So I feel like the folks making the decision to stick with these old "supported" distros are only getting half the story - sure, it won't break, but are they actually getting the security fixes that they think they are? Debian's staff are stretched pretty thin as it is.
In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment?
One other confusion I have about these environments is why they want very-old Python but don't also want very-old Twisted.
Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner.
This is somewhat moot, since I'd be pleasantly surprised if we managed to get another release out at all before September :). But if this is your time frame then lagging your Twisted upgrade by one or two months doesn't seem like it would be worth us supporting this version for much longer.
(FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs)
This is more or less the way I feel about Twisted, particularly around supporting old and weird kernels, because sometimes you really just can't control the environment you're being installed into and I don't want to make Twisted a blocker for that. But supporting old Pythons, old service_identity modules, old OpenSSL's, etc, has been seeming more and more to me like a disservice to the community, because it facilitates the adoption of slow, insecure, dangerous deployment practices so we're not doing anyone any favors. Anyway thank you very much for providing this perspective. I'm curious about how folks deploying ossified software like this think about its security profile, but it's good to know that the practice is still going strong. -glyph
![](https://secure.gravatar.com/avatar/e5a514e14e44913930aa1ac15f508746.jpg?s=120&d=mm&r=g)
On 08/06/2020 08:04, Glyph wrote:
<a bunch of valid stuff>
I'm going to start here by saying: I agree with almost all of what you wrote, but at the end of the day, I don't get to determine our customers' policies. You can try to explain to them why their policies are misguided, but particularly when you're working with a large organisation, change can be very slow. So you end up working around the policy, whether you agree with it or not. In practical terms, that means that for now at least we need to support Python 3.5. As Erik said, we certainly have no right to demand that Twisted continue to support 3.5: indeed, if dropping support will deliver value to the project, then I'd encourage you to go for it; and as you've already said, the whole thing is probably moot anyway given the timescales we're talking about.
I think it's less that they think that Debian does extra vetting, and more that, especially if you're managing whole fleets of servers, then if everything runs the same version, it's easier to keep track of what you need to upgrade when there's a "security" bug. And yes, there are plenty of counterarguments to this, but that's the reasoning.
These are probably all true, but taken to their logical extreme, the conclusion seems to be "you should always run the bleeding edge of all software, to make sure you've got all the latest bug fixes". I don't think you're really arguing for that, so the point is: we end up nominating "stable" versions, and trying to make an assessment as to which bugfixes are worth backporting. That latter part is a subjective decision, and the question is who you trust to make it. You may not trust Debian to make that decision on your behalf (with perfectly valid reasons), but plenty of others do.
A counterpoint here is that the Python in oldstable has had several years of bugfixes, and of course it was the primary Debian-supported version of Python for a good couple of years. Again, I know that peoples' assessment of Debian's ability or competence varies, but I don't think it's *unreasonable* to assume that by this point the worst problems with that version of Python have been shaken out (and that if a significant new problem arises, a fix will be made).
Well, again, it comes down to fleet management, and responsibility for "security". From the customer's point of view, they want to provide a python interpreter which runs our application. So the Python interpreter is their responsibility (hence: use Debian oldstable everywhere), whereas the stuff running on it is ours. Plus, since there's only one thing using Twisted in their network, it's inherently easier to maintain a single version. (I also fear that there is a misguided belief that security vulnerabilities are "worse" if they are in the Python interpreter, because that runs native code, whereas Twisted can't possibly do anything that bad because "interpreted language". I mention this only for completeness, and fully realise it's nonsense.)
Whilst I largely agree with the sentiment, I suspect it's a bit idealistic to take this to an extreme.
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
On Thu, 14 May 2020, at 15:13, Glyph wrote:
A year ago (https://atleastfornow.net/posts/py2-py3-twisted-downloads/), those numbers said that 3.5 was 6.7% of users for the currently released version of Twisted, and 4.37% of all Twisted downloads that month. I could probably rerun them, but I think it's kind of obvious what the results would be.
![](https://secure.gravatar.com/avatar/174e7b0ff60963f821d0b9a4f1a3ef52.jpg?s=120&d=mm&r=g)
On 14. May 2020, at 07:04, Amber Brown (hawkowl) <hawkowl@atleastfornow.net> wrote:
- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7: ❯ /usr/bin/python2 WARNING: Python 2.7 is not recommended. This version is included in macOS for compatibility with legacy software. Future versions of macOS will not include Python 2.7. Instead, it is recommended that you transition to using 'python3' from within Terminal. Python 2.7.16 (default, Feb 29 2020, 01:55:37) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin Type "help", "copyright", "credits" or "license" for more information.
^D
~ took 6s ❯ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Oh hawkowl. Either you die an a.out, or you live long enough to become a .dylib
.... But, it's good to know, anyway.
Unfortunately it's only kind of true! $ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
You can also get one from /Library/Developer/CommandLineTools/usr/bin/python3 if you do `xcode-select --install` but it's not present in the macOS base image and you can't use it from applications. -g
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Wed, May 13, 2020 at 10:05 PM Amber Brown (hawkowl) < hawkowl@atleastfornow.net> wrote:
In trunk, on Python 3.8, all the tests in CI pass on Mac and Linux. With Windows and Python 3.8, there are 6 errors due to AyncioReactorTests failing: https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1771&view=logs&j=ea01aad7-d5f3-5f0e-2fda-71d72e596cac&t=f47a629b-ce24-5ee0-314a-24c887b72299&l=16071 most likely due to this: https://twistedmatrix.com/trac/ticket/9766 where in Python 3.8, the ProactorEventLoop was made the default on Windows. So once this is fixed on Windows, I think we are pretty close to having clean CI runs on Python 3.8 on all platforms. -- Craig
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix). Why I think this is probably fine: - Debian Stable (buster) has 3.7. oldstable (stretch) has 3.5.3, but I don't think we care about oldstable. - Ubuntu 18.04 has Python 3.6. 16.04 has 3.5.1, which is a sucky, buggy version that people shouldn't use anyway. Plus, getting a modern Python on Ubuntu isn't shockingly hard (deadsnakes makes it trivial). - CentOS and RHEL 6 and 7 have Software Collections which makes Python 3.6+ readily available. - Windows has 3.8 available in the Store. - MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily. - For everyone else, there's always Docker. So, I guess if we have 3.8 support, the next release can announce the dropping of 3.5, and then the release after that can be the final one with support. - Amber On 14/5/20 5:10 am, Craig Rodrigues wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) <hawkowl@atleastfornow.net> wrote:
I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5. The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy. -g
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality (due to it being provisional then?) which means things like Eliot doesn't work due to aiocontextvars barfing on missing functionality. There's also 20.04 LTS now, making 16.04 old-old-LTS. I don't think it's worth it. - Amber On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote:
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
Personally, I don't have any dependencies on Python 3.5. I just suggested once year's notice to give people time to adjust, but if no one really cares, dropping 3.5 support some time this year is fine by me. Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support? -- Craig On Wed, May 13, 2020 at 10:37 PM Amber Brown <hawkowl@atleastfornow.net> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.
Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I don't want to be putting (non-trivial numbers of) users into the position of having to skip a bunch of updates. -glyph
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:
Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that work completed by the end of the year. In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure. ---Tom
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Glad to hear it :).
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure.
Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :). -g
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure. Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :).
Oh it's not for lack of wanting. :-) The service in question is actually spawning Python processes, so posix_spawn support doesn't seem likely to help much: fork() + exec() is a drop in the bucket compared to Python imports, particularly in Python 3 (I'm hoping to explore oxidized-importer [1] to help with this at some point). If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so. ---Tom [1]: https://pypi.org/project/oxidized-importer/
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
Would putting all the .pyc/.pyo into a zip file help with the import problem? You only have one file to open, the .zip, and no compile as its only .pyc/.pyo.
Why not waitpid for all children (pid=0 or pid=-1) in a single call? Barry
---Tom
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely? Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!) Anyway there's lot of exciting room for performance improvements here, it would be nice to have some benchmarks for this kind of high-load process-monitor application. -g
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely?
Those all sound like reasonable optimizations with little downside to me.
Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!)
Nonono, that was a synthetic stress test. It did spawn Python processes, but they immediately called `exec sleep` to conserve memory. The result of the test was that the service would run out of memory _well_ before inefficiencies in Twisted's reaping scheme became a problem. ---Tom
![](https://secure.gravatar.com/avatar/e5a514e14e44913930aa1ac15f508746.jpg?s=120&d=mm&r=g)
On 16/05/2020 06:56, Glyph wrote:
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7... I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions? It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy. Or, for that matter, why not just run in a Docker container? Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it? -glyph
![](https://secure.gravatar.com/avatar/89b7125ce22cf40fd72ffebbc54ea8b4.jpg?s=120&d=mm&r=g)
On 23/05/2020 06:39, Glyph wrote:
I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever). Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure. Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode), it might become easier to justify that it really /is/ reasonable for us to require a newer version of Python (and that they should really really upgrade their debian version, and its their own fault that they haven't and have to deal with faff that comes from that). This would also be easier if debian had newer versions of python in backports, or someone ran a semi-official package repository which had them, but as far as I can tell no one does for debian. (Of course you can argue that all the above is a bit silly from a security perspective, especially when you start considering virtualenvs and the like, but I have some sympathy for their outlook even if it is a pain at times). In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment? Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner. (FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs) - Erik
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I think this sounds like a misunderstanding of Debian's vetting process? It's not like there's a ton of additional auditing that goes into packaging something. There's definitely an authentication process for both Twisted and Python, although this attestation could be somewhat stronger and less centralized, PyPI does quite a bit of heavy lifting there.
Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure.
It's worth noting that Python 3.5 itself is out of security support (per https://www.python.org/dev/peps/pep-0478/ <https://www.python.org/dev/peps/pep-0478/>) and it's not like Debian employs a team of Python security-support engineers to continue maintaining this dead branch. If their infrastructure were to try to comply with various security standards such as PCI-DSS, you might bump into the need to upgrade to a newer version with available future patch releases anyway.
Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode),
You say "Autumn", but... oldstable is supported for the purposes of migrating off of it... it's not really even supported by Debian proper. To quote from https://wiki.debian.org/DebianOldStable <https://wiki.debian.org/DebianOldStable> :
The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.
According to https://wiki.debian.org/DebianBuster <https://wiki.debian.org/DebianBuster>, Buster was released on 2019-07-06, which means we've now got less than one month of even this small amount of support that is offered; i.e. the expectation from Debian's perspective is that they'd be done migrating off of oldstable by that point. I guess https://wiki.debian.org/LTS <https://wiki.debian.org/LTS> is a separate project?
It's a bit of a pet peeve of mine for reasons which are mostly covered by this blog post -https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/ <https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/> But it bugs me not just because of the expectation of free labor (we can refuse to support 3.5 if that is a point of contention, as you say) but because "security support" is a bit of a misnomer, especially in Debian's case where there's no commercial relationship and hence not much in the way of libability. Much to my chagrin I can't find the paper I remember reading about this right now, but, I believe these things are all true: Many non-"security" bugs are in fact security bugs that nobody has noticed you can exploit. Many "low-severity" or "un-exploiable" security bugs are in fact exploitable "supported" distros rarely take care to backport many patches for their software, and when they do, they often make undetected errors (like debian's infamous ssh bug) which are analyzed by far fewer security analysts than the upstream source code. So I feel like the folks making the decision to stick with these old "supported" distros are only getting half the story - sure, it won't break, but are they actually getting the security fixes that they think they are? Debian's staff are stretched pretty thin as it is.
In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment?
One other confusion I have about these environments is why they want very-old Python but don't also want very-old Twisted.
Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner.
This is somewhat moot, since I'd be pleasantly surprised if we managed to get another release out at all before September :). But if this is your time frame then lagging your Twisted upgrade by one or two months doesn't seem like it would be worth us supporting this version for much longer.
(FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs)
This is more or less the way I feel about Twisted, particularly around supporting old and weird kernels, because sometimes you really just can't control the environment you're being installed into and I don't want to make Twisted a blocker for that. But supporting old Pythons, old service_identity modules, old OpenSSL's, etc, has been seeming more and more to me like a disservice to the community, because it facilitates the adoption of slow, insecure, dangerous deployment practices so we're not doing anyone any favors. Anyway thank you very much for providing this perspective. I'm curious about how folks deploying ossified software like this think about its security profile, but it's good to know that the practice is still going strong. -glyph
![](https://secure.gravatar.com/avatar/e5a514e14e44913930aa1ac15f508746.jpg?s=120&d=mm&r=g)
On 08/06/2020 08:04, Glyph wrote:
<a bunch of valid stuff>
I'm going to start here by saying: I agree with almost all of what you wrote, but at the end of the day, I don't get to determine our customers' policies. You can try to explain to them why their policies are misguided, but particularly when you're working with a large organisation, change can be very slow. So you end up working around the policy, whether you agree with it or not. In practical terms, that means that for now at least we need to support Python 3.5. As Erik said, we certainly have no right to demand that Twisted continue to support 3.5: indeed, if dropping support will deliver value to the project, then I'd encourage you to go for it; and as you've already said, the whole thing is probably moot anyway given the timescales we're talking about.
I think it's less that they think that Debian does extra vetting, and more that, especially if you're managing whole fleets of servers, then if everything runs the same version, it's easier to keep track of what you need to upgrade when there's a "security" bug. And yes, there are plenty of counterarguments to this, but that's the reasoning.
These are probably all true, but taken to their logical extreme, the conclusion seems to be "you should always run the bleeding edge of all software, to make sure you've got all the latest bug fixes". I don't think you're really arguing for that, so the point is: we end up nominating "stable" versions, and trying to make an assessment as to which bugfixes are worth backporting. That latter part is a subjective decision, and the question is who you trust to make it. You may not trust Debian to make that decision on your behalf (with perfectly valid reasons), but plenty of others do.
A counterpoint here is that the Python in oldstable has had several years of bugfixes, and of course it was the primary Debian-supported version of Python for a good couple of years. Again, I know that peoples' assessment of Debian's ability or competence varies, but I don't think it's *unreasonable* to assume that by this point the worst problems with that version of Python have been shaken out (and that if a significant new problem arises, a fix will be made).
Well, again, it comes down to fleet management, and responsibility for "security". From the customer's point of view, they want to provide a python interpreter which runs our application. So the Python interpreter is their responsibility (hence: use Debian oldstable everywhere), whereas the stuff running on it is ours. Plus, since there's only one thing using Twisted in their network, it's inherently easier to maintain a single version. (I also fear that there is a misguided belief that security vulnerabilities are "worse" if they are in the Python interpreter, because that runs native code, whereas Twisted can't possibly do anything that bad because "interpreted language". I mention this only for completeness, and fully realise it's nonsense.)
Whilst I largely agree with the sentiment, I suspect it's a bit idealistic to take this to an extreme.
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
On Thu, 14 May 2020, at 15:13, Glyph wrote:
A year ago (https://atleastfornow.net/posts/py2-py3-twisted-downloads/), those numbers said that 3.5 was 6.7% of users for the currently released version of Twisted, and 4.37% of all Twisted downloads that month. I could probably rerun them, but I think it's kind of obvious what the results would be.
![](https://secure.gravatar.com/avatar/174e7b0ff60963f821d0b9a4f1a3ef52.jpg?s=120&d=mm&r=g)
On 14. May 2020, at 07:04, Amber Brown (hawkowl) <hawkowl@atleastfornow.net> wrote:
- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7: ❯ /usr/bin/python2 WARNING: Python 2.7 is not recommended. This version is included in macOS for compatibility with legacy software. Future versions of macOS will not include Python 2.7. Instead, it is recommended that you transition to using 'python3' from within Terminal. Python 2.7.16 (default, Feb 29 2020, 01:55:37) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin Type "help", "copyright", "credits" or "license" for more information.
^D
~ took 6s ❯ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Oh hawkowl. Either you die an a.out, or you live long enough to become a .dylib
.... But, it's good to know, anyway.
Unfortunately it's only kind of true! $ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
You can also get one from /Library/Developer/CommandLineTools/usr/bin/python3 if you do `xcode-select --install` but it's not present in the macOS base image and you can't use it from applications. -g
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Wed, May 13, 2020 at 10:05 PM Amber Brown (hawkowl) < hawkowl@atleastfornow.net> wrote:
In trunk, on Python 3.8, all the tests in CI pass on Mac and Linux. With Windows and Python 3.8, there are 6 errors due to AyncioReactorTests failing: https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1771&view=logs&j=ea01aad7-d5f3-5f0e-2fda-71d72e596cac&t=f47a629b-ce24-5ee0-314a-24c887b72299&l=16071 most likely due to this: https://twistedmatrix.com/trac/ticket/9766 where in Python 3.8, the ProactorEventLoop was made the default on Windows. So once this is fixed on Windows, I think we are pretty close to having clean CI runs on Python 3.8 on all platforms. -- Craig
participants (11)
-
Amber Brown
-
Amber Brown (hawkowl)
-
Barry Scott
-
Craig Rodrigues
-
Erik Johnston
-
Glyph
-
Hynek Schlawack
-
Jean-Paul Calderone
-
Moshe Zadka
-
Richard van der Hoff
-
Tom Most