May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, so maybe we can wait? I don't have strong feelings, but it is a datapoint. Moshe Z. On Wed, May 13, 2020, at 22:13, Glyph wrote:

On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) wrote:

...

On 14/5/20 5:10 am, Craig Rodrigues wrote:

...

I would like to propose that Twisted drop support for Python 3.5 one year from now in 2021.

I was considering proposing dropping it soon (I was thinking when we had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).

I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.

The lack of variable annotations, in particular, seems like a sticking point with 3.5 as we attempt to adopt mypy.

-g

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Personally, I don't have any dependencies on Python 3.5. I just suggested once year's notice to give people time to adjust, but if no one really cares, dropping 3.5 support some time this year is fine by me. Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support? -- Craig On Wed, May 13, 2020 at 10:37 PM Amber Brown wrote:

Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality (due to it being provisional then?) which means things like Eliot doesn't work due to aiocontextvars barfing on missing functionality. There's also 20.04 LTS now, making 16.04 old-old-LTS. I don't think it's worth it.

- Amber

...

May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, so maybe we can wait? I don't have strong feelings, but it is a datapoint.

Moshe Z.

On Wed, May 13, 2020, at 22:13, Glyph wrote:

...

On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) < hawkowl@atleastfornow.net> wrote:

...

On 14/5/20 5:10 am, Craig Rodrigues wrote:

...

I would like to propose that Twisted drop support for Python 3.5

one year from now in 2021.

...

I was considering proposing dropping it soon (I was thinking when we

had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).

I'm also inclined to say sooner rather than later; I don't know the numbers but my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, so there aren't that many people still on 3.5.

The lack of variable annotations, in particular, seems like a sticking

On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote: point with 3.5 as we attempt to adopt mypy.

...

...

-g

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars. ---Tom On Fri, May 15, 2020, at 10:56 PM, Glyph wrote:

...

On May 15, 2020, at 8:40 PM, Craig Rodrigues wrote:

Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support? Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.

Any 3.5 users on this list who would want to postpone it longer than this?

-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:

...

On May 16, 2020, at 3:12 PM, Tom Most wrote: I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 3.5. However, I am fine with dropping Python 3.5 support. I think it's certainly worth it for variable annotations and contextvars.

Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I don't want to be putting (non-trivial numbers of) users into the position of having to skip a bunch of updates.

-glyph

Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that work completed by the end of the year. In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure. ---Tom

On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:

...

In the one case that may be complicated I think that pinning Twisted will be acceptable, since the service in question is a process supervisor without direct external exposure. Well in that case I'll just do my best to add lots of goodies (posix_spawn support!) that will make you want to upgrade that to current twisted and python 3.8 :).

Oh it's not for lack of wanting. :-) The service in question is actually spawning Python processes, so posix_spawn support doesn't seem likely to help much: fork() + exec() is a drop in the bucket compared to Python imports, particularly in Python 3 (I'm hoping to explore oxidized-importer [1] to help with this at some point). If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so. ---Tom [1]: https://pypi.org/project/oxidized-importer/

On Tue, May 19, 2020 at 8:33 AM Barry Scott wrote:

On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:

...

If looking improve process management in Twisted, I think that the lowest-hanging fruit is to watch children with pidfd, rather than reaping all children on SIGCHLD. This O(n) reaping --- one waitpid() call per running process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 processes or so.

Why not waitpid for all children (pid=0 or pid=-1) in a single call?

Barry

Because then you might reap a child process someone launched with a different API - eg the Python subprocess module. Jean-Paul

On May 19, 2020, at 11:14 AM, Tom Most wrote:

On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:

...

On Tue, May 19, 2020 at 8:33 AM Barry Scott mailto:barry.scott@forcepoint.com> wrote: Why not waitpid for all children (pid=0 or pid=-1) in a single call?

Because then you might reap a child process someone launched with a different API - eg the Python subprocess module.

Exactly. I am currently using this as a workaround, but it's not a general solution. You need to be careful because subprocess use hides in many places. For example, beware uuid.uuid1().

So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely? Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!) Anyway there's lot of exciting room for performance improvements here, it would be nice to have some benchmarks for this kind of high-load process-monitor application. -g

On May 28, 2020, at 3:13 PM, Tom Most wrote:

On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:

...

So, pidfd's cool, we should totally use it. Also we should use posix_spawn and maybe some other stuff too. But I wonder if there's any heuristic we could use to speed up our current strategy, like ordering the to-reap list by putting things with no open FDs at the front of it? And optimistically assuming that once we've found something to reap, maybe we can stop? And maybe it should run in a cooperator, rather than just blocking the reactor indefinitely?

Those all sound like reasonable optimizations with little downside to me.

Great! Anybody feel like filing some tickets? :)

...

Honestly it had not occurred to me that people were managing 20k+ python interpreters at a time with spawnProcess (although, yikes, amazing, you've gotta talk more about this application, and what kind of hardware you're on!)

Nonono, that was a synthetic stress test. It did spawn Python processes, but they immediately called `exec sleep` to conserve memory.

The result of the test was that the service would run out of memory _well_ before inefficiencies in Twisted's reaping scheme became a problem.

Aah, that's less exciting ;). -g

On 16/05/2020 06:56, Glyph wrote:

...

On May 15, 2020, at 8:40 PM, Craig Rodrigues mailto:rodrigc@crodrigues.org> wrote:

Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?

Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.

Any 3.5 users on this list who would want to postpone it longer than this?

Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7... I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.

On 23/05/2020 06:39, Glyph wrote:

...

On May 19, 2020, at 1:52 AM, Richard van der Hoff mailto:richard@matrix.org> wrote:

On 16/05/2020 06:56, Glyph wrote:

...

...

On May 15, 2020, at 8:40 PM, Craig Rodrigues mailto:rodrigc@crodrigues.org> wrote:

Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?

Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.

Any 3.5 users on this list who would want to postpone it longer than this?

Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7...

I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.

I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions?  It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy.  Or, for that matter, why not just run in a Docker container?

Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it?

-glyph

I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever). Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure. Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode), it might become easier to justify that it really /is/ reasonable for us to require a newer version of Python (and that they should really really upgrade their debian version, and its their own fault that they haven't and have to deal with faff that comes from that). This would also be easier if debian had newer versions of python in backports, or someone ran a semi-official package repository which had them, but as far as I can tell no one does for debian. (Of course you can argue that all the above is a bit silly from a security perspective, especially when you start considering virtualenvs and the like, but I have some sympathy for their outlook even if it is a pain at times). In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment? Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner. (FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs) - Erik

On 08/06/2020 08:04, Glyph wrote:

<a bunch of valid stuff>

I'm going to start here by saying: I agree with almost all of what you wrote, but at the end of the day, I don't get to determine our customers' policies. You can try to explain to them why their policies are misguided, but particularly when you're working with a large organisation, change can be very slow. So you end up working around the policy, whether you agree with it or not. In practical terms, that means that for now at least we need to support Python 3.5. As Erik said, we certainly have no right to demand that Twisted continue to support 3.5: indeed, if dropping support will deliver value to the project, then I'd encourage you to go for it; and as you've already said, the whole thing is probably moot anyway given the timescales we're talking about.

...

I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever).

I think this sounds like a misunderstanding of Debian's vetting process?  It's not like there's a ton of additional auditing that goes into packaging something.  There's definitely an authentication process for both Twisted and Python, although this attestation could be somewhat stronger and less centralized, PyPI does quite a bit of heavy lifting there.

I think it's less that they think that Debian does extra vetting, and more that, especially if you're managing whole fleets of servers, then if everything runs the same version, it's easier to keep track of what you need to upgrade when there's a "security" bug. And yes, there are plenty of counterarguments to this, but that's the reasoning.

1. Many non-"security" bugs are in fact security bugs that nobody has noticed you can exploit. 2. Many "low-severity" or "un-exploiable" security bugs are in fact exploitable 3. "supported" distros rarely take care to backport many patches for their software, and when they do, they often make undetected errors (like debian's infamous ssh bug) which are analyzed by far fewer security analysts than the upstream source code.

These are probably all true, but taken to their logical extreme, the conclusion seems to be "you should always run the bleeding edge of all software, to make sure you've got all the latest bug fixes". I don't think you're really arguing for that, so the point is: we end up nominating "stable" versions, and trying to make an assessment as to which bugfixes are worth backporting. That latter part is a subjective decision, and the question is who you trust to make it. You may not trust Debian to make that decision on your behalf (with perfectly valid reasons), but plenty of others do.

So I feel like the folks making the decision to stick with these old "supported" distros are only getting half the story - sure, it won't break, but are they /actually/ getting the security fixes that they think they are?  Debian's staff are stretched pretty thin as it is.

A counterpoint here is that the Python in oldstable has had several years of bugfixes, and of course it was the primary Debian-supported version of Python for a good couple of years. Again, I know that peoples' assessment of Debian's ability or competence varies, but I don't think it's *unreasonable* to assume that by this point the worst problems with that version of Python have been shaken out (and that if a significant new problem arises, a fix will be made).

...

In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment?

One other confusion I have about these environments is why they want very-old Python but don't /also/ want very-old Twisted.

Well, again, it comes down to fleet management, and responsibility for "security". From the customer's point of view, they want to provide a python interpreter which runs our application. So the Python interpreter is their responsibility (hence: use Debian oldstable everywhere), whereas the stuff running on it is ours. Plus, since there's only one thing using Twisted in their network, it's inherently easier to maintain a single version. (I also fear that there is a misguided belief that security vulnerabilities are "worse" if they are in the Python interpreter, because that runs native code, whereas Twisted can't possibly do anything that bad because "interpreted language". I mention this only for completeness, and fully realise it's nonsense.)

But supporting old Pythons, old service_identity modules, old OpenSSL's, etc, has been seeming more and more to me like a disservice to the community, because it facilitates the adoption of slow, insecure, dangerous deployment practices so we're not doing anyone any favors.

Whilst I largely agree with the sentiment, I suspect it's a bit idealistic to take this to an extreme.

On 14. May 2020, at 07:04, Amber Brown (hawkowl) wrote:

- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.

FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7: ❯ /usr/bin/python2 WARNING: Python 2.7 is not recommended. This version is included in macOS for compatibility with legacy software. Future versions of macOS will not include Python 2.7. Instead, it is recommended that you transition to using 'python3' from within Terminal. Python 2.7.16 (default, Feb 29 2020, 01:55:37) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin Type "help", "copyright", "credits" or "license" for more information.

...

...

^D

~ took 6s ❯ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.

On May 13, 2020, at 11:13 PM, Amber Brown wrote:

On Thu, 14 May 2020, at 16:09, Hynek Schlawack wrote:

...

...

On 14. May 2020, at 07:04, Amber Brown (hawkowl) wrote:

- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.

FWIW this is not true anymore. Catalina comes – for all its faults – with Python 3.7:

*puts fingers in ears* sorry I can't hear you, I'm in my happy place where I don't have to think of software in ".framework" folders la la la la

Oh hawkowl. Either you die an a.out, or you live long enough to become a .dylib

.... But, it's good to know, anyway.

Unfortunately it's only kind of true! $ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.

...

...

import sys sys.executable '/Applications/Xcode.app/Contents/Developer/usr/bin/python3'

You can also get one from /Library/Developer/CommandLineTools/usr/bin/python3 if you do `xcode-select --install` but it's not present in the macOS base image and you can't use it from applications. -g

On May 14, 2020, at 3:55 AM, Amber Brown wrote:

On Thu, 14 May 2020, at 16:54, Glyph wrote:

...

...

$ /usr/bin/python3 Python 3.7.3 (default, Apr 7 2020, 14:06:47) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import sys sys.executable '/Applications/Xcode.app/Contents/Developer/usr/bin/python3'

wow thanks I hate it

ITYM 'Wow thanks I hate it.app' -g