[Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.
Reasons:
1. Python 3.5.9 will be the final release of Python 3.5, in November 2019 ( https://www.python.org/dev/peps/pep-0478/ ) 2. Python 3.6.0 was released in December 2016 ( https://www.python.org/dev/peps/pep-0494/ ) 3. PyPy 7.1.1 introduced Python 3.6 support in April 2019 ( https://morepypy.blogspot.com/2019/04/pypy-711-bug-fix-release.html ). PyPy 7.3.1 is the latest PyPy version ( https://morepypy.blogspot.com/2020/04/pypy-731-released.html ).
4. Python 3.6 introduced support for variable annotations ( https://www.python.org/dev/peps/pep-0526/ ) 5. Python 3.6 introduced new string interpolation (f-string) support ( https://www.python.org/dev/peps/pep-0498/ ) 6. Python 3.6 introduced support for asynchronous generators ( https://www.python.org/dev/peps/pep-0525/ ) and asynchronous comprehensions ( https://www.python.org/dev/peps/pep-0530/ )
Based on item 2., Python 3.6.0 has been around for almost 4 years, which is a reasonable amount of time. Based on item 3., PyPy supports Python 3.6.
It might be nice to take advantage of some of the language features introduces in items 4-6.
I think dropping support one year from now in 2021, would give sufficient notice, and be in line with the Twisted Compatibility Policy ( https://twistedmatrix.com/documents/current/core/development/policy/compatib... ).
None of these items are critial, but I thought I would ask, to get people's opinions.
-- Craig
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
Why I think this is probably fine:
- Debian Stable (buster) has 3.7. oldstable (stretch) has 3.5.3, but I don't think we care about oldstable. - Ubuntu 18.04 has Python 3.6. 16.04 has 3.5.1, which is a sucky, buggy version that people shouldn't use anyway. Plus, getting a modern Python on Ubuntu isn't shockingly hard (deadsnakes makes it trivial). - CentOS and RHEL 6 and 7 have Software Collections which makes Python 3.6+ readily available. - Windows has 3.8 available in the Store. - MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily. - For everyone else, there's always Docker.
So, I guess if we have 3.8 support, the next release can announce the dropping of 3.5, and then the release after that can be the final one with support.
- Amber
On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.
Reasons:
1. Python 3.5.9 will be the final release of Python 3.5, in November 2019 ( https://www.python.org/dev/peps/pep-0478/ ) 2. Python 3.6.0 was released in December 2016 ( https://www.python.org/dev/peps/pep-0494/ ) 3. PyPy 7.1.1 introduced Python 3.6 support in April 2019 ( https://morepypy.blogspot.com/2019/04/pypy-711-bug-fix-release.html ). PyPy 7.3.1 is the latest PyPy version ( https://morepypy.blogspot.com/2020/04/pypy-731-released.html ).
4. Python 3.6 introduced support for variable annotations ( https://www.python.org/dev/peps/pep-0526/ ) 5. Python 3.6 introduced new string interpolation (f-string) support ( https://www.python.org/dev/peps/pep-0498/ ) 6. Python 3.6 introduced support for asynchronous generators ( https://www.python.org/dev/peps/pep-0525/ ) and asynchronous comprehensions ( https://www.python.org/dev/peps/pep-0530/ )
Based on item 2., Python 3.6.0 has been around for almost 4 years, which is a reasonable amount of time. Based on item 3., PyPy supports Python 3.6.
It might be nice to take advantage of some of the language features introduces in items 4-6.
I think dropping support one year from now in 2021, would give sufficient notice, and be in line with the Twisted Compatibility Policy ( https://twistedmatrix.com/documents/current/core/development/policy/compatib... ).
None of these items are critial, but I thought I would ask, to get people's opinions.
-- Craig
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.
The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy.
-g
May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, so maybe we can wait? I don't have strong feelings, but it is a datapoint.
Moshe Z.
On Wed, May 13, 2020, at 22:13, Glyph wrote:
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.
The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy.
-g
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality (due to it being provisional then?) which means things like Eliot doesn't work due to aiocontextvars barfing on missing functionality. There's also 20.04 LTS now, making 16.04 old-old-LTS. I don't think it's worth it.
- Amber
On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote:
May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, so maybe we can wait? I don't have strong feelings, but it is a datapoint.
Moshe Z.
On Wed, May 13, 2020, at 22:13, Glyph wrote:
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.
The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy.
-g
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Personally, I don't have any dependencies on Python 3.5. I just suggested once year's notice to give people time to adjust, but if no one really cares, dropping 3.5 support some time this year is fine by me.
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support? -- Craig
On Wed, May 13, 2020 at 10:37 PM Amber Brown hawkowl@atleastfornow.net wrote:
Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality (due to it being provisional then?) which means things like Eliot doesn't work due to aiocontextvars barfing on missing functionality. There's also 20.04 LTS now, making 16.04 old-old-LTS. I don't think it's worth it.
- Amber
On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote:
May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, so maybe we can wait? I don't have strong feelings, but it is a datapoint.
Moshe Z.
On Wed, May 13, 2020, at 22:13, Glyph wrote:
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) <
hawkowl@atleastfornow.net> wrote:
On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5
one year from now in 2021.
I was considering proposing dropping it soon (I was thinking when we
had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
I'm also inclined to say sooner rather than later; I don't know the
numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.
The lack of variable annotations, in particular, seems like a sticking
point with 3.5 as we attempt to adopt mypy.
-g
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On May 15, 2020, at 8:40 PM, Craig Rodrigues rodrigc@crodrigues.org wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
-glyph
I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.
---Tom
On Fri, May 15, 2020, at 10:56 PM, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues rodrigc@crodrigues.org wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On May 16, 2020, at 3:12 PM, Tom Most twm@freecog.net wrote:
On Fri, May 15, 2020, at 10:56 PM, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues <rodrigc@crodrigues.org mailto:rodrigc@crodrigues.org> wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
-glyph
I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.
Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I don't want to be putting (non-trivial numbers of) users into the position of having to skip a bunch of updates.
-glyph
On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:
On May 16, 2020, at 3:12 PM, Tom Most twm@freecog.net wrote: I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.
Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I don't want to be putting (non-trivial numbers of) users into the position of having to skip a bunch of updates.
-glyph
Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that work completed by the end of the year.
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure.
---Tom
On May 17, 2020, at 12:10 PM, Tom Most twm@freecog.net wrote:
On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:
On May 16, 2020, at 3:12 PM, Tom Most <twm@freecog.net mailto:twm@freecog.net> wrote: I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.
Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I don't want to be putting (non-trivial numbers of) users into the position of having to skip a bunch of updates.
-glyph
Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that work completed by the end of the year.
Glad to hear it :).
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure.
Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :).
-g
On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure.
Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :).
Oh it's not for lack of wanting. :-)
The service in question is actually spawning Python processes, so posix_spawn support doesn't seem likely to help much: fork() + exec() is a drop in the bucket compared to Python imports, particularly in Python 3 (I'm hoping to explore oxidized-importer [1] to help with this at some point).
If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so.
---Tom
On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure.>
Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :).
Oh it's not for lack of wanting. :-)
The service in question is actually spawning Python processes, so posix_spawn support doesn't seem likely to help much: fork() + exec() is a drop in the bucket compared to Python imports, particularly in Python 3 (I'm hoping to explore oxidized-importer [1] to help with this at some point).
Would putting all the .pyc/.pyo into a zip file help with the import problem? You only have one file to open, the .zip, and no compile as its only .pyc/.pyo.
If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so.
Why not waitpid for all children (pid=0 or pid=-1) in a single call?
Barry
---Tom
On Tue, May 19, 2020 at 8:33 AM Barry Scott barry.scott@forcepoint.com wrote:
On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so.
Why not waitpid for all children (pid=0 or pid=-1) in a single call?
Barry
Because then you might reap a child process someone launched with a different API - eg the Python subprocess module.
Jean-Paul
On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:
On Tue, May 19, 2020 at 8:33 AM Barry Scott barry.scott@forcepoint.com wrote:
Why not waitpid for all children (pid=0 or pid=-1) in a single call?
Because then you might reap a child process someone launched with a different API - eg the Python subprocess module.
Exactly. I am currently using this as a workaround, but it's not a general solution. You need to be careful because subprocess use hides in many places. For example, beware uuid.uuid1().
---Tom
On May 19, 2020, at 11:14 AM, Tom Most twm@freecog.net wrote:
On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:
On Tue, May 19, 2020 at 8:33 AM Barry Scott <barry.scott@forcepoint.com mailto:barry.scott@forcepoint.com> wrote: Why not waitpid for all children (pid=0 or pid=-1) in a single call?
Because then you might reap a child process someone launched with a different API - eg the Python subprocess module.
Exactly. I am currently using this as a workaround, but it's not a general solution. You need to be careful because subprocess use hides in many places. For example, beware uuid.uuid1().
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely? Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!)
Anyway there's lot of exciting room for performance improvements here, it would be nice to have some benchmarks for this kind of high-load process-monitor application.
-g
On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely?
Those all sound like reasonable optimizations with little downside to me.
Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!)
Nonono, that was a synthetic stress test. It did spawn Python processes, but they immediately called `exec sleep` to conserve memory.
The result of the test was that the service would run out of memory _well_ before inefficiencies in Twisted's reaping scheme became a problem.
---Tom
On May 28, 2020, at 3:13 PM, Tom Most twm@freecog.net wrote:
On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:
So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely?
Those all sound like reasonable optimizations with little downside to me.
Great! Anybody feel like filing some tickets? :)
Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!)
Nonono, that was a synthetic stress test. It did spawn Python processes, but they immediately called `exec sleep` to conserve memory.
The result of the test was that the service would run out of memory _well_ before inefficiencies in Twisted's reaping scheme became a problem.
Aah, that's less exciting ;).
-g
On May 19, 2020, at 5:41 AM, Jean-Paul Calderone exarkun@twistedmatrix.com wrote:
On Tue, May 19, 2020 at 8:33 AM Barry Scott <barry.scott@forcepoint.com mailto:barry.scott@forcepoint.com> wrote: On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so.
Why not waitpid for all children (pid=0 or pid=-1) in a single call?
Barry
Because then you might reap a child process someone launched with a different API - eg the Python subprocess module.
In case it is not clear from context - we used to do exactly this. It broke a lot of stuff. The current approach is slow, but robust.
-g
On 16/05/2020 06:56, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues <rodrigc@crodrigues.org mailto:rodrigc@crodrigues.org> wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7...
I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
On May 19, 2020, at 1:52 AM, Richard van der Hoff richard@matrix.org wrote:
On 16/05/2020 06:56, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues <rodrigc@crodrigues.org mailto:rodrigc@crodrigues.org> wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7...
I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions? It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy. Or, for that matter, why not just run in a Docker container?
Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it?
-glyph
On 23/05/2020 06:39, Glyph wrote:
On May 19, 2020, at 1:52 AM, Richard van der Hoff <richard@matrix.org mailto:richard@matrix.org> wrote:
On 16/05/2020 06:56, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues <rodrigc@crodrigues.org mailto:rodrigc@crodrigues.org> wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7...
I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions? It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy. Or, for that matter, why not just run in a Docker container?
Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it?
-glyph
I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever).
Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure.
Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode), it might become easier to justify that it really /is/ reasonable for us to require a newer version of Python (and that they should really really upgrade their debian version, and its their own fault that they haven't and have to deal with faff that comes from that).
This would also be easier if debian had newer versions of python in backports, or someone ran a semi-official package repository which had them, but as far as I can tell no one does for debian.
(Of course you can argue that all the above is a bit silly from a security perspective, especially when you start considering virtualenvs and the like, but I have some sympathy for their outlook even if it is a pain at times).
In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment? Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner.
(FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs)
- Erik
On May 28, 2020, at 8:31 AM, Erik Johnston erik@matrix.org wrote:
On 23/05/2020 06:39, Glyph wrote:
On May 19, 2020, at 1:52 AM, Richard van der Hoff <richard@matrix.org mailto:richard@matrix.org> wrote:
On 16/05/2020 06:56, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues <rodrigc@crodrigues.org mailto:rodrigc@crodrigues.org> wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7...
I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions? It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy. Or, for that matter, why not just run in a Docker container?
Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it?
-glyph
I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever).
I think this sounds like a misunderstanding of Debian's vetting process? It's not like there's a ton of additional auditing that goes into packaging something. There's definitely an authentication process for both Twisted and Python, although this attestation could be somewhat stronger and less centralized, PyPI does quite a bit of heavy lifting there.
Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure.
It's worth noting that Python 3.5 itself is out of security support (per https://www.python.org/dev/peps/pep-0478/ https://www.python.org/dev/peps/pep-0478/) and it's not like Debian employs a team of Python security-support engineers to continue maintaining this dead branch. If their infrastructure were to try to comply with various security standards such as PCI-DSS, you might bump into the need to upgrade to a newer version with available future patch releases anyway.
Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode),
You say "Autumn", but... oldstable is supported for the purposes of migrating off of it... it's not really even supported by Debian proper. To quote from https://wiki.debian.org/DebianOldStable https://wiki.debian.org/DebianOldStable :
The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.
According to https://wiki.debian.org/DebianBuster https://wiki.debian.org/DebianBuster, Buster was released on 2019-07-06, which means we've now got less than one month of even this small amount of support that is offered; i.e. the expectation from Debian's perspective is that they'd be done migrating off of oldstable by that point. I guess https://wiki.debian.org/LTS https://wiki.debian.org/LTS is a separate project?
it might become easier to justify that it really is reasonable for us to require a newer version of Python (and that they should really really upgrade their debian version, and its their own fault that they haven't and have to deal with faff that comes from that).
This would also be easier if debian had newer versions of python in backports, or someone ran a semi-official package repository which had them, but as far as I can tell no one does for debian.
(Of course you can argue that all the above is a bit silly from a security perspective, especially when you start considering virtualenvs and the like, but I have some sympathy for their outlook even if it is a pain at times).
It's a bit of a pet peeve of mine for reasons which are mostly covered by this blog post -https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/ https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/
But it bugs me not just because of the expectation of free labor (we can refuse to support 3.5 if that is a point of contention, as you say) but because "security support" is a bit of a misnomer, especially in Debian's case where there's no commercial relationship and hence not much in the way of libability.
Much to my chagrin I can't find the paper I remember reading about this right now, but, I believe these things are all true:
Many non-"security" bugs are in fact security bugs that nobody has noticed you can exploit. Many "low-severity" or "un-exploiable" security bugs are in fact exploitable "supported" distros rarely take care to backport many patches for their software, and when they do, they often make undetected errors (like debian's infamous ssh bug) which are analyzed by far fewer security analysts than the upstream source code.
So I feel like the folks making the decision to stick with these old "supported" distros are only getting half the story - sure, it won't break, but are they actually getting the security fixes that they think they are? Debian's staff are stretched pretty thin as it is.
In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment?
One other confusion I have about these environments is why they want very-old Python but don't also want very-old Twisted.
Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner.
This is somewhat moot, since I'd be pleasantly surprised if we managed to get another release out at all before September :). But if this is your time frame then lagging your Twisted upgrade by one or two months doesn't seem like it would be worth us supporting this version for much longer.
(FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs)
This is more or less the way I feel about Twisted, particularly around supporting old and weird kernels, because sometimes you really just can't control the environment you're being installed into and I don't want to make Twisted a blocker for that. But supporting old Pythons, old service_identity modules, old OpenSSL's, etc, has been seeming more and more to me like a disservice to the community, because it facilitates the adoption of slow, insecure, dangerous deployment practices so we're not doing anyone any favors.
Anyway thank you very much for providing this perspective. I'm curious about how folks deploying ossified software like this think about its security profile, but it's good to know that the practice is still going strong.
-glyph
On 08/06/2020 08:04, Glyph wrote:
<a bunch of valid stuff>
I'm going to start here by saying: I agree with almost all of what you wrote, but at the end of the day, I don't get to determine our customers' policies. You can try to explain to them why their policies are misguided, but particularly when you're working with a large organisation, change can be very slow. So you end up working around the policy, whether you agree with it or not. In practical terms, that means that for now at least we need to support Python 3.5.
As Erik said, we certainly have no right to demand that Twisted continue to support 3.5: indeed, if dropping support will deliver value to the project, then I'd encourage you to go for it; and as you've already said, the whole thing is probably moot anyway given the timescales we're talking about.
I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever).
I think this sounds like a misunderstanding of Debian's vetting process? It's not like there's a ton of additional auditing that goes into packaging something. There's definitely an authentication process for both Twisted and Python, although this attestation could be somewhat stronger and less centralized, PyPI does quite a bit of heavy lifting there.
I think it's less that they think that Debian does extra vetting, and more that, especially if you're managing whole fleets of servers, then if everything runs the same version, it's easier to keep track of what you need to upgrade when there's a "security" bug. And yes, there are plenty of counterarguments to this, but that's the reasoning.
- Many non-"security" bugs are in fact security bugs that nobody has noticed you can exploit.
- Many "low-severity" or "un-exploiable" security bugs are in fact exploitable
- "supported" distros rarely take care to backport many patches for their software, and when they do, they often make undetected errors (like debian's infamous ssh bug) which are analyzed by far fewer security analysts than the upstream source code.
These are probably all true, but taken to their logical extreme, the conclusion seems to be "you should always run the bleeding edge of all software, to make sure you've got all the latest bug fixes". I don't think you're really arguing for that, so the point is: we end up nominating "stable" versions, and trying to make an assessment as to which bugfixes are worth backporting. That latter part is a subjective decision, and the question is who you trust to make it. You may not trust Debian to make that decision on your behalf (with perfectly valid reasons), but plenty of others do.
So I feel like the folks making the decision to stick with these old "supported" distros are only getting half the story - sure, it won't break, but are they /actually/ getting the security fixes that they think they are? Debian's staff are stretched pretty thin as it is.
A counterpoint here is that the Python in oldstable has had several years of bugfixes, and of course it was the primary Debian-supported version of Python for a good couple of years. Again, I know that peoples' assessment of Debian's ability or competence varies, but I don't think it's *unreasonable* to assume that by this point the worst problems with that version of Python have been shaken out (and that if a significant new problem arises, a fix will be made).
In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment?
One other confusion I have about these environments is why they want very-old Python but don't /also/ want very-old Twisted.
Well, again, it comes down to fleet management, and responsibility for "security". From the customer's point of view, they want to provide a python interpreter which runs our application. So the Python interpreter is their responsibility (hence: use Debian oldstable everywhere), whereas the stuff running on it is ours. Plus, since there's only one thing using Twisted in their network, it's inherently easier to maintain a single version.
(I also fear that there is a misguided belief that security vulnerabilities are "worse" if they are in the Python interpreter, because that runs native code, whereas Twisted can't possibly do anything that bad because "interpreted language". I mention this only for completeness, and fully realise it's nonsense.)
But supporting old Pythons, old service_identity modules, old OpenSSL's, etc, has been seeming more and more to me like a disservice to the community, because it facilitates the adoption of slow, insecure, dangerous deployment practices so we're not doing anyone any favors.
Whilst I largely agree with the sentiment, I suspect it's a bit idealistic to take this to an extreme.
On Thu, 14 May 2020, at 15:13, Glyph wrote:
On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.
I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.
A year ago (https://atleastfornow.net/posts/py2-py3-twisted-downloads/), those numbers said that 3.5 was 6.7% of users for the currently released version of Twisted, and 4.37% of all Twisted downloads that month. I could probably rerun them, but I think it's kind of obvious what the results would be.
The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy.
-g
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 14. May 2020, at 07:04, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7:
❯ /usr/bin/python2
WARNING: Python 2.7 is not recommended. This version is included in macOS for compatibility with legacy software. Future versions of macOS will not include Python 2.7. Instead, it is recommended that you transition to using 'python3' from within Terminal.
Python 2.7.16 (default, Feb 29 2020, 01:55:37) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin Type "help", "copyright", "credits" or "license" for more information.
^D
~ took 6s ❯ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
On Thu, 14 May 2020, at 16:09, Hynek Schlawack wrote:
On 14. May 2020, at 07:04, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7:
*puts fingers in ears* sorry I can't hear you, I'm in my happy place where I don't have to think of software in ".framework" folders la la la la
.... But, it's good to know, anyway.
- Amber
On May 13, 2020, at 11:13 PM, Amber Brown hawkowl@atleastfornow.net wrote:
On Thu, 14 May 2020, at 16:09, Hynek Schlawack wrote:
On 14. May 2020, at 07:04, Amber Brown (hawkowl) hawkowl@atleastfornow.net wrote:
- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7:
*puts fingers in ears* sorry I can't hear you, I'm in my happy place where I don't have to think of software in ".framework" folders la la la la
Oh hawkowl. Either you die an a.out, or you live long enough to become a .dylib
.... But, it's good to know, anyway.
Unfortunately it's only kind of true!
$ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import sys sys.executable
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
You can also get one from /Library/Developer/CommandLineTools/usr/bin/python3 if you do `xcode-select --install` but it's not present in the macOS base image and you can't use it from applications.
-g
On Thu, 14 May 2020, at 16:54, Glyph wrote:
$ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import sys sys.executable
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
wow thanks I hate it
-g
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On May 14, 2020, at 3:55 AM, Amber Brown hawkowl@atleastfornow.net wrote:
On Thu, 14 May 2020, at 16:54, Glyph wrote:
$ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import sys sys.executable
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
wow thanks I hate it
ITYM 'Wow thanks I hate it.app'
-g
On Wed, May 13, 2020 at 10:05 PM Amber Brown (hawkowl) < hawkowl@atleastfornow.net> wrote:
So, I guess if we have 3.8 support, the next release can announce the dropping of 3.5, and then the release after that can be the final one with support.
In trunk, on Python 3.8, all the tests in CI pass on Mac and Linux. With Windows and Python 3.8, there are 6 errors due to AyncioReactorTests failing: https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1771&...
most likely due to this: https://twistedmatrix.com/trac/ticket/9766
where in Python 3.8, the ProactorEventLoop was made the default on Windows.
So once this is fixed on Windows, I think we are pretty close to having clean CI runs on Python 3.8 on all platforms. -- Craig
participants (11)
-
Amber Brown -
Amber Brown (hawkowl)
-
Barry Scott -
Craig Rodrigues -
Erik Johnston -
Glyph -
Hynek Schlawack -
Jean-Paul Calderone -
Moshe Zadka -
Richard van der Hoff -
Tom Most