[Twisted-Python] A Proposal for reducing the burden of developing on Twisted by dropping Python 2 support
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
Hi everyone, Since the Python 2 EOL date is rapidly approaching, I thought it was time we consider dropping Python 2 support. I personally find that Python 2 compat adds a huge amount of overhead when working on and maintaining Twisted, and think that with the current maintainer availability, dropping it sooner rather than later would have a beneficial effect on how much work we spend on shims/compat, complexity, and our ability to ship new features, as well as onboarding people who are interested in the project, but have no interest (or experience!) in Python 2.7. It is basically summed up by doing a feature freeze on an agreed-upon version of Twisted, that will be the last version released for 2.7. It would be abnormal in that it would get security fixes (our current policy is to only release them for current versions) and critical bugfixes, but would otherwise be frozen. One of my rationales is that from some analysis of PyPI download statistics, the vast majority of Python 2 users are using old versions of Twisted, while nearly all our Python 3 users are on the latest version. As such, I believe freezing a version that will get security updates but no new features would not be a massive loss to those stuck on Python 2 for whatever reason. Twisted's compatibility policy would still apply, ensuring that Python 2/3 compatible software using Twisted would be able to use the older Twisted version on Python 2, and the newer version on Python 3, as you would usually expect. You can find the proposal here, in this handy-dandy Google Doc: https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0... Comments are welcome, as well as which timeline seems reasonable. - Amber
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Monday, 25 March 2019 11:15:22 CET Amber Brown wrote:
I think this makes sense: applications that are still gaining new features should be on Python 3 by now. For applications on Python 2 the main concern would be to keep them running while the world around them updates (new OS versions etc).
Comments are welcome, as well as which timeline seems reasonable.
Python 2.7 support will end early 2020. In my opinion it is unnecessary to support Twisted on Python 2.7 for longer than Python 2.7 itself is supported. So all of the proposed timelines are generous, in my opinion. Bye, Maarten
![](https://secure.gravatar.com/avatar/1ab02b123e4645acb4a53fe974d1742b.jpg?s=120&d=mm&r=g)
Hi Amber - Notwithstanding the fact that VFX/Anim will be stuck on PY2 for a few more years, this all sounds very reasonable. None of the proposed timelines would cause problems for my team. Many thanks DJM On Mon, 25 Mar 2019 at 10:15, Amber Brown <hawkowl@atleastfornow.net> wrote:
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Monday, 25 March 2019 10:15:22 GMT Amber Brown wrote:
We are stuck on python2.7 for the foreseeable future and you will not see us in the PyPi stats. We download your release tar ball and build an RPM from it. We also have to build python2.7 our selves as we are stuck on Centos 6.8. Having a legacy twisted with critical bugs and security fixes would work for us. Barry
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Hi Barry! Thanks for your feedback.
Given the nature of your product I'm curious: isn't the inability to maintain certifications like PCI / HIPAA due to the lack of upstream support for python 2.7 a problem for a security application?
Having a legacy twisted with critical bugs and security fixes would work for us.
Apropos of the question in my other response - would you (or any of your colleagues) be willing to take responsibility for such a branch and do the security triage / backport work there? -g
![](https://secure.gravatar.com/avatar/8285ea1765ae52a3ab79c767676a2f4e.jpg?s=120&d=mm&r=g)
My application works on both Python 2.7 and 3. All my and my customer sites are now on Python 3 (there was a year or so of both), so I've stopped testing on 2.7, though I haven't done anything deliberate to break it. I hope/expect that none of my code changes to make my source compatible with both 2.7 and 3 will break when Twisted support for 2.7 goes away! (It seems to me the Twisted compatibility policy would guarantee that.) Most of my customer systems are on Red Hat or CentOS where the default Python, as of a few weeks ago, is 2.7. I've installed Python 3.6 in a virtual environment and that has worked fine. (Replacing 2.7 with 3.6 breaks many system tools, don't ask me how I know this!) So I no longer have any problem with 2.7 support going away in Twisted. - John Santos On 3/25/2019 6:15 AM, Amber Brown wrote:
-- John Santos Evans Griffiths & Hart, Inc. 781-861-0670 ext 539
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Personally, I don't have this problem, but I'm certainly willing to believe it's a bigger deal for others; especially others who perhaps have not even learned Python 2 at this point.
It is basically summed up by doing a feature freeze on an agreed-upon version of Twisted, that will be the last version released for 2.7. It would be abnormal in that it would get security fixes (our current policy is to only release them for current versions) and critical bugfixes, but would otherwise be frozen.
Here's my question about this: Who will do this work? Personally, I'm not willing to commit to this. I know from experience both on Twisted and other projects that maintaining multiple release branches, even one that's "maintenance only", requires at least one point-person for each branch at all times (usually a "release manager"). And backporting fixes inevitably gets harder and harder as the "maintenance" branch diverges from top-of-tree. If I have time to work on Twisted in my increasingly scarce spare time, I want it to be on something at least plausibly interesting, and manually backporting manual fixes to an unmaintained py2 branch that I don't even use doesn't qualify. ("But glyph", I hear you ask, "then why are you willing to write py2 code as it is?". Well, hypothetical reader, it's because I find it easily an order of magnitude easier to fix version discrepancies in the same change where they creep in, and CI is yelling me about them, than it is to fix them in a big tangled morass six months later with no idea which part of the code I'm backporting is the issue.) Do we have py2-only users who are willing to take ownership of this branch? Specific people willing to sign up for this responsibility for the next... how long? Three years? Five years? So I'd rather be quite explicit that while we would not object to anyone filling these roles, someone still needs to step forward and fill them, and I'm not willing to commit the current team, such as it is, to work that I myself am unwilling to do. If nobody does step forward we should not claim to have security support for a dead / unmaintained branch.
One of my rationales is that from some analysis of PyPI download statistics, the vast majority of Python 2 users are using old versions of Twisted, while nearly all our Python 3 users are on the latest version. As such, I believe freezing a version that will get security updates but no new features would not be a massive loss to those stuck on Python 2 for whatever reason.
I know that you detailed some of this on IRC, but: how old? My sense would be that of course users stuck on py2 would have a more conservative upgrade cadence than py3 users, but that doesn't mean they never upgrade. How far behind are these py2 users, and does the curve suggest they're catching up or are most twisted downloads just like, version 1.3 on python 1.5.2 forever?
Twisted's compatibility policy would still apply, ensuring that Python 2/3 compatible software using Twisted would be able to use the older Twisted version on Python 2, and the newer version on Python 3, as you would usually expect.
We have 2 big blockers here right now that would prevent doing this as things stand right now: Twisted is not yet fully ported to Python 3, so there's no version where you can use all of Twisted on Python 3. This picture is way, way smaller than it has ever been, though - grab a module and start porting: http://blog.habnab.it/twisted-depgraph/ <http://blog.habnab.it/twisted-depgraph/> We still have Python 2 in production ourselves that requires Twisted; specifically: Trac and several of our ancillary utilities around it. Happily this does seem to be fairly actively worked on: "opened 4 years ago, last modified 12 days ago" https://trac.edgewall.org/ticket/12130 <https://trac.edgewall.org/ticket/12130> Dogfood DNS: twisted.names doesn't work on python 3: https://twistedmatrix.com/trac/ticket/9496 <https://twistedmatrix.com/trac/ticket/9496> Our front-end webserver might work on pypy3, but we have yet to move it over. Maybe that would be a good place to start? If this proposal lights a fire under some folks to drive any of these projects to completion, that would be great!
You can find the proposal here, in this handy-dandy Google Doc: https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0...
Comments are welcome, as well as which timeline seems reasonable.
Personally I feel like Option 1 is the most reasonable. We can't do option 3 because of the above blockers, and option 2 just seems random to me - why commit to 4 months of additional maintenance beyond when py2 itself is EOL? Maybe there's an option like, 1.5, where we de-support py2 at max(2020-01-01, "the date at which we no longer have any py2 code in production ourselves")? -g
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
On Tue, 26 Mar 2019 at 04:52, Glyph <glyph@twistedmatrix.com> wrote:
I am still using python 2.7 on all my projects, and I remember the sending patches supporting python 3 was an extra effort... I had to learn python 3 things, and then search for ways / workaround to implement something in a py2 + py3 way... read six documentation. I guess that many Twisted developers don't have too much free time, this can add up. For me learning py3 was ok as at some point I will use it...but I guess that for people who will never use py2 is just unwanted work.
[SNIP]
Do we have py2-only users who are willing to take ownership of this branch? Specific people willing to sign up for this responsibility for the next... how long? Three years? Five years?
I can help with backporting security fixed for the next 5 years, as I will use it. I am still using 15.5 due to lack of python cryptography on all systems, and I have a fork for twisted.web with support for resources which can add custom `Expect: 100-continue` behaviour. I am doing backporting and as long as there is a CI, is not that hard. I hope there will be at least another person to help reviewing the backports.
So I'd rather be quite explicit that while we would not object to anyone filling these roles, someone still needs to step forward and fill them, and I'm not willing to commit the current team, such as it is, to work that I myself am unwilling to do. If nobody does step forward we should not claim to have security support for a dead / unmaintained branch.
Well.. this is free/libre software so I guess that nobody will be sued if we commit to something, and later we found out that this can't be done.
PCI don't help. So far, they have only asked for TLS1.2 and removal of CBC Mode ciphers in SSH...and that is all. They have never complained about old OpenSSH or OpenSSL versions... and when you have custom old software, the PCI auditor have no clue. I know more than 3 big companies which still have not upgraded to TLS 1.2 even with all the PCI audits...they always get extensions and workarounds.
Do we really need Trac? Maybe is time to move the tickets and wiki to GitHub.
Dogfood DNS: twisted.names doesn't work on python 3: https://twistedmatrix.com/trac/ticket/9496
I have never used Twisted.names in production. I tried to help with the review of twisted.names patches, and I remember there was not much interest for twisted.names So maybe we can get rid of it and move it to a separate project. I don't think that we need to have everything in twisted core... if ldaptor is an external project, why we can't have twisted.names
Our front-end webserver might work on pypy3, but we have yet to move it over. Maybe that would be a good place to start?
If this proposal lights a fire under some folks to drive any of these projects to completion, that would be great!
My hope is that this proposal will light a fire and people using Twisted will more about Twisted. I feel that Twisted stability and reduce number of defects is not helping the open source dev model. People will use Twisted, it just work and they forget about it. Or if it doesn't work, they can fix something on their project relatively easy...but takes a lot of effort to send upstream..so they will never put the effort required to have a fix upstream.
If we keep the current prod infrastructure, I am not confident that we will no longer have py2 code in prod by the end of this year. I don't feel that the current Twisted team have resource to maintain a custom infrastructure. I think that things will be better for everyone if for example of Twisted names we will just use the credit on Azure to keep the DNS and use more Software as a Service so that the team will spend more time on developing new feature than doing sysadmin work. Dogfooding is a good thing, but in this case, I fell that it does not help. -- Adi Roiban
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Mon, Mar 25, 2019 at 3:15 AM Amber Brown <hawkowl@atleastfornow.net> wrote:
You can find the proposal here, in this handy-dandy Google Doc:
https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0...
Thanks for writing that doc. I think option 3 of making Twisted 19.4 the last Python 2 release, with security updates until 2021 is actually pretty reasonable. -- Craig
![](https://secure.gravatar.com/avatar/d8bc8f87492826483cda8faa4386b30d.jpg?s=120&d=mm&r=g)
I don't have any skin in this game other than as a consumer, but I agree that the proposed plan makes sense, FWIW. Freezing at a specific version in sync (ish) with the Py2 support notification has a certain symmetry to it :) Regards, Jeff
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Monday, 25 March 2019 11:15:22 CET Amber Brown wrote:
I think this makes sense: applications that are still gaining new features should be on Python 3 by now. For applications on Python 2 the main concern would be to keep them running while the world around them updates (new OS versions etc).
Comments are welcome, as well as which timeline seems reasonable.
Python 2.7 support will end early 2020. In my opinion it is unnecessary to support Twisted on Python 2.7 for longer than Python 2.7 itself is supported. So all of the proposed timelines are generous, in my opinion. Bye, Maarten
![](https://secure.gravatar.com/avatar/1ab02b123e4645acb4a53fe974d1742b.jpg?s=120&d=mm&r=g)
Hi Amber - Notwithstanding the fact that VFX/Anim will be stuck on PY2 for a few more years, this all sounds very reasonable. None of the proposed timelines would cause problems for my team. Many thanks DJM On Mon, 25 Mar 2019 at 10:15, Amber Brown <hawkowl@atleastfornow.net> wrote:
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Monday, 25 March 2019 10:15:22 GMT Amber Brown wrote:
We are stuck on python2.7 for the foreseeable future and you will not see us in the PyPi stats. We download your release tar ball and build an RPM from it. We also have to build python2.7 our selves as we are stuck on Centos 6.8. Having a legacy twisted with critical bugs and security fixes would work for us. Barry
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Hi Barry! Thanks for your feedback.
Given the nature of your product I'm curious: isn't the inability to maintain certifications like PCI / HIPAA due to the lack of upstream support for python 2.7 a problem for a security application?
Having a legacy twisted with critical bugs and security fixes would work for us.
Apropos of the question in my other response - would you (or any of your colleagues) be willing to take responsibility for such a branch and do the security triage / backport work there? -g
![](https://secure.gravatar.com/avatar/8285ea1765ae52a3ab79c767676a2f4e.jpg?s=120&d=mm&r=g)
My application works on both Python 2.7 and 3. All my and my customer sites are now on Python 3 (there was a year or so of both), so I've stopped testing on 2.7, though I haven't done anything deliberate to break it. I hope/expect that none of my code changes to make my source compatible with both 2.7 and 3 will break when Twisted support for 2.7 goes away! (It seems to me the Twisted compatibility policy would guarantee that.) Most of my customer systems are on Red Hat or CentOS where the default Python, as of a few weeks ago, is 2.7. I've installed Python 3.6 in a virtual environment and that has worked fine. (Replacing 2.7 with 3.6 breaks many system tools, don't ask me how I know this!) So I no longer have any problem with 2.7 support going away in Twisted. - John Santos On 3/25/2019 6:15 AM, Amber Brown wrote:
-- John Santos Evans Griffiths & Hart, Inc. 781-861-0670 ext 539
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Personally, I don't have this problem, but I'm certainly willing to believe it's a bigger deal for others; especially others who perhaps have not even learned Python 2 at this point.
It is basically summed up by doing a feature freeze on an agreed-upon version of Twisted, that will be the last version released for 2.7. It would be abnormal in that it would get security fixes (our current policy is to only release them for current versions) and critical bugfixes, but would otherwise be frozen.
Here's my question about this: Who will do this work? Personally, I'm not willing to commit to this. I know from experience both on Twisted and other projects that maintaining multiple release branches, even one that's "maintenance only", requires at least one point-person for each branch at all times (usually a "release manager"). And backporting fixes inevitably gets harder and harder as the "maintenance" branch diverges from top-of-tree. If I have time to work on Twisted in my increasingly scarce spare time, I want it to be on something at least plausibly interesting, and manually backporting manual fixes to an unmaintained py2 branch that I don't even use doesn't qualify. ("But glyph", I hear you ask, "then why are you willing to write py2 code as it is?". Well, hypothetical reader, it's because I find it easily an order of magnitude easier to fix version discrepancies in the same change where they creep in, and CI is yelling me about them, than it is to fix them in a big tangled morass six months later with no idea which part of the code I'm backporting is the issue.) Do we have py2-only users who are willing to take ownership of this branch? Specific people willing to sign up for this responsibility for the next... how long? Three years? Five years? So I'd rather be quite explicit that while we would not object to anyone filling these roles, someone still needs to step forward and fill them, and I'm not willing to commit the current team, such as it is, to work that I myself am unwilling to do. If nobody does step forward we should not claim to have security support for a dead / unmaintained branch.
One of my rationales is that from some analysis of PyPI download statistics, the vast majority of Python 2 users are using old versions of Twisted, while nearly all our Python 3 users are on the latest version. As such, I believe freezing a version that will get security updates but no new features would not be a massive loss to those stuck on Python 2 for whatever reason.
I know that you detailed some of this on IRC, but: how old? My sense would be that of course users stuck on py2 would have a more conservative upgrade cadence than py3 users, but that doesn't mean they never upgrade. How far behind are these py2 users, and does the curve suggest they're catching up or are most twisted downloads just like, version 1.3 on python 1.5.2 forever?
Twisted's compatibility policy would still apply, ensuring that Python 2/3 compatible software using Twisted would be able to use the older Twisted version on Python 2, and the newer version on Python 3, as you would usually expect.
We have 2 big blockers here right now that would prevent doing this as things stand right now: Twisted is not yet fully ported to Python 3, so there's no version where you can use all of Twisted on Python 3. This picture is way, way smaller than it has ever been, though - grab a module and start porting: http://blog.habnab.it/twisted-depgraph/ <http://blog.habnab.it/twisted-depgraph/> We still have Python 2 in production ourselves that requires Twisted; specifically: Trac and several of our ancillary utilities around it. Happily this does seem to be fairly actively worked on: "opened 4 years ago, last modified 12 days ago" https://trac.edgewall.org/ticket/12130 <https://trac.edgewall.org/ticket/12130> Dogfood DNS: twisted.names doesn't work on python 3: https://twistedmatrix.com/trac/ticket/9496 <https://twistedmatrix.com/trac/ticket/9496> Our front-end webserver might work on pypy3, but we have yet to move it over. Maybe that would be a good place to start? If this proposal lights a fire under some folks to drive any of these projects to completion, that would be great!
You can find the proposal here, in this handy-dandy Google Doc: https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0...
Comments are welcome, as well as which timeline seems reasonable.
Personally I feel like Option 1 is the most reasonable. We can't do option 3 because of the above blockers, and option 2 just seems random to me - why commit to 4 months of additional maintenance beyond when py2 itself is EOL? Maybe there's an option like, 1.5, where we de-support py2 at max(2020-01-01, "the date at which we no longer have any py2 code in production ourselves")? -g
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
On Tue, 26 Mar 2019 at 04:52, Glyph <glyph@twistedmatrix.com> wrote:
I am still using python 2.7 on all my projects, and I remember the sending patches supporting python 3 was an extra effort... I had to learn python 3 things, and then search for ways / workaround to implement something in a py2 + py3 way... read six documentation. I guess that many Twisted developers don't have too much free time, this can add up. For me learning py3 was ok as at some point I will use it...but I guess that for people who will never use py2 is just unwanted work.
[SNIP]
Do we have py2-only users who are willing to take ownership of this branch? Specific people willing to sign up for this responsibility for the next... how long? Three years? Five years?
I can help with backporting security fixed for the next 5 years, as I will use it. I am still using 15.5 due to lack of python cryptography on all systems, and I have a fork for twisted.web with support for resources which can add custom `Expect: 100-continue` behaviour. I am doing backporting and as long as there is a CI, is not that hard. I hope there will be at least another person to help reviewing the backports.
So I'd rather be quite explicit that while we would not object to anyone filling these roles, someone still needs to step forward and fill them, and I'm not willing to commit the current team, such as it is, to work that I myself am unwilling to do. If nobody does step forward we should not claim to have security support for a dead / unmaintained branch.
Well.. this is free/libre software so I guess that nobody will be sued if we commit to something, and later we found out that this can't be done.
PCI don't help. So far, they have only asked for TLS1.2 and removal of CBC Mode ciphers in SSH...and that is all. They have never complained about old OpenSSH or OpenSSL versions... and when you have custom old software, the PCI auditor have no clue. I know more than 3 big companies which still have not upgraded to TLS 1.2 even with all the PCI audits...they always get extensions and workarounds.
Do we really need Trac? Maybe is time to move the tickets and wiki to GitHub.
Dogfood DNS: twisted.names doesn't work on python 3: https://twistedmatrix.com/trac/ticket/9496
I have never used Twisted.names in production. I tried to help with the review of twisted.names patches, and I remember there was not much interest for twisted.names So maybe we can get rid of it and move it to a separate project. I don't think that we need to have everything in twisted core... if ldaptor is an external project, why we can't have twisted.names
Our front-end webserver might work on pypy3, but we have yet to move it over. Maybe that would be a good place to start?
If this proposal lights a fire under some folks to drive any of these projects to completion, that would be great!
My hope is that this proposal will light a fire and people using Twisted will more about Twisted. I feel that Twisted stability and reduce number of defects is not helping the open source dev model. People will use Twisted, it just work and they forget about it. Or if it doesn't work, they can fix something on their project relatively easy...but takes a lot of effort to send upstream..so they will never put the effort required to have a fix upstream.
If we keep the current prod infrastructure, I am not confident that we will no longer have py2 code in prod by the end of this year. I don't feel that the current Twisted team have resource to maintain a custom infrastructure. I think that things will be better for everyone if for example of Twisted names we will just use the credit on Azure to keep the DNS and use more Software as a Service so that the team will spend more time on developing new feature than doing sysadmin work. Dogfooding is a good thing, but in this case, I fell that it does not help. -- Adi Roiban
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Mon, Mar 25, 2019 at 3:15 AM Amber Brown <hawkowl@atleastfornow.net> wrote:
You can find the proposal here, in this handy-dandy Google Doc:
https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0...
Thanks for writing that doc. I think option 3 of making Twisted 19.4 the last Python 2 release, with security updates until 2021 is actually pretty reasonable. -- Craig
![](https://secure.gravatar.com/avatar/d8bc8f87492826483cda8faa4386b30d.jpg?s=120&d=mm&r=g)
I don't have any skin in this game other than as a consumer, but I agree that the proposed plan makes sense, FWIW. Freezing at a specific version in sync (ish) with the Py2 support notification has a certain symmetry to it :) Regards, Jeff
participants (11)
-
Adi Roiban
-
Amber Brown
-
Craig Rodrigues
-
Donal McMullan
-
Glyph
-
Jeff Grimmett
-
John Santos
-
Maarten ter Huurne
-
Scott, Barry
-
Tom Most
-
Werner Thie