On May 28, 2020, at 8:31 AM, Erik Johnston <erik@matrix.org> wrote:
On 23/05/2020 06:39, Glyph wrote:
On May 19, 2020, at 1:52 AM, Richard van der Hoff <richard@matrix.org <mailto:richard@matrix.org>> wrote:
On 16/05/2020 06:56, Glyph wrote:
On May 15, 2020, at 8:40 PM, Craig Rodrigues <rodrigc@crodrigues.org <mailto:rodrigc@crodrigues.org>> wrote:
Maybe it would be OK to do one more release of Twisted and announce that as the last release supporting Python 3.5, before dropping support?
Yeah; whenever we drop a Python version we should always support at least one more release, so that people have some notice before they lose access to the next set of security updates.
Any 3.5 users on this list who would want to postpone it longer than this?
Sadly we have an important customer whose servers run debian oldstable, which means we need to stay compatible with 3.5 until we can persuade them to upgrade, and it's taken a couple of years to get them off python 2.7...
I'm not sure that should necessarily affect your plans, but I doubt we're alone in this situation.
I guess one thing I'm curious about is why your application would need to be installed along with the system Python on those OS versions? It seems like a packaging strategy that ignored the fossilized versions that Debian packages with the system and just built its own Python would be more reliable and allow for upgrading at least most Python dependencies well beyond what the system would allow by policy. Or, for that matter, why not just run in a Docker container?
Matrix is a pretty big user, and so in some sense I care about this specific case, but I also find the general question interesting, because I have difficulty reasoning about how long to support older versions of things in the modern application packaging environment where containers, virtualenvs, and associated tooling make it possible to effectively ignore the base environment. When & why do you have to pay attention to it?
-glyph
I believe in this case its a general desire to keep track of what packages are running and where they've come from. They basically trust that packages from official Debian repositories are probably safe from being tampered with, whereas random tarballs of code from the web are not safe (unless they're signed by someone they trust or whatever).
I think this sounds like a misunderstanding of Debian's vetting process? It's not like there's a ton of additional auditing that goes into packaging something. There's definitely an authentication process for both Twisted and Python, although this attestation could be somewhat stronger and less centralized, PyPI does quite a bit of heavy lifting there.
Now, I think it would be possible to get a newer version of Python on their infrastructure if we needed, but I'm sure there would be hoops that would need to be jumped through and justifications given, etc, which would undoubtedly take some time. So really it just means extra faff for them and us, especially since we're only a small part of their overall infrastructure.
It's worth noting that Python 3.5 itself is out of security support (per https://www.python.org/dev/peps/pep-0478/ <https://www.python.org/dev/peps/pep-0478/>) and it's not like Debian employs a team of Python security-support engineers to continue maintaining this dead branch. If their infrastructure were to try to comply with various security standards such as PCI-DSS, you might bump into the need to upgrade to a newer version with available future patch releases anyway.
Then there is the fact that they're not unique. While oldstable is, well, old, its still very much supported and so there's going to be a bunch of "enterprise" (for want of a better term) customers who will still be using it, and we'll need to go through the faff each and every time, which is quite tedious. Come the Autumn when oldstable stops being supported (or at least, goes into LTS mode),
You say "Autumn", but... oldstable is supported for the purposes of migrating off of it... it's not really even supported by Debian proper. To quote from https://wiki.debian.org/DebianOldStable <https://wiki.debian.org/DebianOldStable> :
The security team tries to support a stable distribution for about one year after the next stable distribution has been released, except when another stable distribution is released within this year. It is not possible to support three distributions; supporting two simultaneously is already difficult enough.
According to https://wiki.debian.org/DebianBuster <https://wiki.debian.org/DebianBuster>, Buster was released on 2019-07-06, which means we've now got less than one month of even this small amount of support that is offered; i.e. the expectation from Debian's perspective is that they'd be done migrating off of oldstable by that point. I guess https://wiki.debian.org/LTS <https://wiki.debian.org/LTS> is a separate project?
it might become easier to justify that it really is reasonable for us to require a newer version of Python (and that they should really really upgrade their debian version, and its their own fault that they haven't and have to deal with faff that comes from that).
This would also be easier if debian had newer versions of python in backports, or someone ran a semi-official package repository which had them, but as far as I can tell no one does for debian.
(Of course you can argue that all the above is a bit silly from a security perspective, especially when you start considering virtualenvs and the like, but I have some sympathy for their outlook even if it is a pain at times).
It's a bit of a pet peeve of mine for reasons which are mostly covered by this blog post -https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/ <https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/> But it bugs me not just because of the expectation of free labor (we can refuse to support 3.5 if that is a point of contention, as you say) but because "security support" is a bit of a misnomer, especially in Debian's case where there's no commercial relationship and hence not much in the way of libability. Much to my chagrin I can't find the paper I remember reading about this right now, but, I believe these things are all true: Many non-"security" bugs are in fact security bugs that nobody has noticed you can exploit. Many "low-severity" or "un-exploiable" security bugs are in fact exploitable "supported" distros rarely take care to backport many patches for their software, and when they do, they often make undetected errors (like debian's infamous ssh bug) which are analyzed by far fewer security analysts than the upstream source code. So I feel like the folks making the decision to stick with these old "supported" distros are only getting half the story - sure, it won't break, but are they actually getting the security fixes that they think they are? Debian's staff are stretched pretty thin as it is.
In terms of twisted dropping support of 3.5, I guess the question is to what extent do you want applications to be hassle free to deploy on the more "enterprise" style environment?
One other confusion I have about these environments is why they want very-old Python but don't also want very-old Twisted.
Without having followed along with the thread my bias would be to keep support until stretch support is ended in the Autumn. Though if 3.5 support is holding things back a lot then I can completely understand dropping it sooner.
This is somewhat moot, since I'd be pleasantly surprised if we managed to get another release out at all before September :). But if this is your time frame then lagging your Twisted upgrade by one or two months doesn't seem like it would be worth us supporting this version for much longer.
(FWIW in Matrix/Synapse we take the view that we retain compat with old-but-supported dependencies, such as postgres, until/unless it starts being too costly in terms of maintenance and opportunity costs)
This is more or less the way I feel about Twisted, particularly around supporting old and weird kernels, because sometimes you really just can't control the environment you're being installed into and I don't want to make Twisted a blocker for that. But supporting old Pythons, old service_identity modules, old OpenSSL's, etc, has been seeming more and more to me like a disservice to the community, because it facilitates the adoption of slow, insecure, dangerous deployment practices so we're not doing anyone any favors. Anyway thank you very much for providing this perspective. I'm curious about how folks deploying ossified software like this think about its security profile, but it's good to know that the practice is still going strong. -glyph