(Looking for) A Retrospective on the Move to Python 3

In pondering our approach to future Python major releases, I found myself considering the experience we've had with Python 3. The whole Py3k effort predates my involvement in the community so I missed a bunch of context about the motivations, decisions, and challenges. While I've pieced some of that together over the years now since I've been around, I've certainly seen much of the aftermath. For me, at least, it would be helpful to have a bit more insight into the history. :) With that in mind, it would be worth having an informational PEP with an authoritative retrospective on the lessons learned from the Python 3 effort (and transition). Consider it a sort of autobiography, "memoirs on the python-dev change to Python 3". :) At this point the transition has settled in enough that we should be able to present a relatively objective (and consistent) view, while we're not so far removed that we've forgotten anything important. :) If such a document already exists then I'd love a pointer to it. The document would benefit (among others): * python-dev (by giving us a clear viewpoint to inform decisions about future releases) * new-comers to Python that want more insight into the language * folks transitioning from 2 to 3 * communities that have (or think they have) problems similar to those we faced in Python 2 The PEP doesn't even have to be done all at once, nor by one person. In fact, there are many viewpoints that would add value to the document. Hence it would probably make sense to encourage broad participation and then have a single editor to effect a single voice in the document. The contents of the retrospective document should probably cover a broad range of topics, since there's so much to learn from the move to Python 3. To give an indication of what I mean, I've included a rough outline at the bottom of this message. So...I typically strongly avoid making proposals that I'm not willing to execute. However, in this case I simply do not have enough experience in the history to feel comfortable doing a good job of it in a reasonable amount of time (which matters due to the tendency of valuable info to fade away). :/ I have no expectation that someone will pick this up, though I do hope since the benefit would be significant. My apologies in advance if this wasted anyone's time. -eric ++++++++++++++++++++++++++++++++ I'd hope to see something along the lines of (at least) the following, in rough order: * a concise summary of the document at the top (very meta, I know :) ) + what were we solving? + what was the solution? + why do it that way? + what went right? + what went wrong? + impact on the community + impact on core dev contribution * timeline * key players (and level of involvement) + old guard core devs + new guard + folks brought on for Py3k (e.g. IIRC a swarm of Googlers dove in) + non-core-devs * motivations * expectations (e.g. time frames, community reaction) * corresponding results * a summary of what we did * alternative approaches * what went right (and was it on purpose :) ) * what went wrong (e.g. io) and why * how the Py3k project differed from normal python-dev workflow (e.g. pace, decision-making, communications) * lasting impact of python-dev * key things that would have been better if done differently * key decisions/planning (mostly a priori to the release work) + scope of backward compatibility + process (using PEPs with PEPs 30xx guiding) + schedule + specific changes (i.e. PEPs 31xx) + what was left out (and why) + plans to help library and app authors transition (e.g. 2to3) + feature/schedule overlap with Python 2 (i.e. 2.6 and 2.7) + the language moratorium * things that got missed and why + unicode/bytes in some stdlib modules (and builtins?) * things that were overdone (and how that got missed) + unicode/bytes in some stdlib modules (and builtins?) * (last but not least) challenges faced by folks working to transition their exiting code to Python 3

On Thu, Apr 26, 2018 at 10:25 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
As to motivation, I suppose (since I wasn't involved yet) that it was roughly: let's bite the bullet and fix unicode with a backward-incompatible release which led to: well, we're already going to break backward compatibility so we might as well get any breakage we're planning over with now so folks only have to fix their code for this release and then to: oh, and while we're at it let's clean up a bunch of cruft that's built up over the years At least that's my impression. :) -eric

On Apr 26, 2018, at 09:28, Eric Snow <ericsnowcurrently@gmail.com> wrote:
It would certainly be an interesting document, but I suspect you’ll get a bit of the old “ask 3 lawyers and get 5 opinions” kind of response. ;) As I remember it, there was definitely a feeling like, this would be our only chance to clean up some annoying cruft, and rectify some (in hindsight) incorrect design decisions made over the years, couple with a healthy dose of “we have no idea how to do the bytes/str split in a backward compatible way". There was probably a sense that the Python community was just small enough to be able to handle such a disruptive change, but wouldn’t ever be so again. The latter is definitely true today, even if the former was overly optimistic. -Barry

On Thu, 26 Apr 2018 at 10:19 Barry Warsaw <barry@python.org> wrote:
I agree with everything Barry said. There are some lessons in hindsight of how we could have handled bytes/str, but it was more of a decision of "really long transition versus a short one" -- jokes on us for what "short" became ;) -- which we simply won't make ever again.

http://pyvideo.org/pycascades-2018/bdfl-python-3-retrospective.html link to Guido’s talk, for your convenience From: Python-Dev <python-dev-bounces+tritium-list=sdamon.com@python.org> On Behalf Of Guido van Rossum Sent: Thursday, April 26, 2018 6:12 PM To: Brett Cannon <brett@python.org> Cc: Barry Warsaw <barry@python.org>; Python-Dev <python-dev@python.org> Subject: Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3 Also see my talk at PyCascades and Victor's upcoming talk at PyCon. On Thu, Apr 26, 2018, 12:02 Brett Cannon <brett@python.org <mailto:brett@python.org> > wrote: On Thu, 26 Apr 2018 at 10:19 Barry Warsaw <barry@python.org <mailto:barry@python.org> > wrote: On Apr 26, 2018, at 09:28, Eric Snow <ericsnowcurrently@gmail.com <mailto:ericsnowcurrently@gmail.com> > wrote:
It would certainly be an interesting document, but I suspect you’ll get a bit of the old “ask 3 lawyers and get 5 opinions” kind of response. ;) As I remember it, there was definitely a feeling like, this would be our only chance to clean up some annoying cruft, and rectify some (in hindsight) incorrect design decisions made over the years, couple with a healthy dose of “we have no idea how to do the bytes/str split in a backward compatible way". There was probably a sense that the Python community was just small enough to be able to handle such a disruptive change, but wouldn’t ever be so again. The latter is definitely true today, even if the former was overly optimistic. I agree with everything Barry said. There are some lessons in hindsight of how we could have handled bytes/str, but it was more of a decision of "really long transition versus a short one" -- jokes on us for what "short" became ;) -- which we simply won't make ever again. _______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

On 04/26/2018 09:02 PM, Alex Walters wrote:
http://pyvideo.org/pycascades-2018/bdfl-python-3-retrospective.html link to Guido’s talk, for your convenience
Many thanks! -- ~Ethan~

On 27 April 2018 at 03:18, Barry Warsaw <barry@python.org> wrote:
http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an... covers some of the questions Eric is asking (mostly from my PoV, but Guido corrected my answer to the initial "Why was Python 3 made incompatible with Python 2?" question shortly after I posted the first version of it). https://www.curiousefficiency.org/posts/2014/08/python-4000.html is a more retrospective-y article that looks more at the implications for Python 4. For the "What actually happened?" info, probably the 3 main documents to look at would be PEP 3000 (the process doc), PEP 3100 (accepted changes that didn't get their own PEPs), and PEP 3099 (explicitly rejected ideas that also didn't get their own PEPs). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I gave the talk "Python 3: 10 years later" at FOSDEM and Pycon Italy and will give it again at Pycon US next month: https://fosdem.org/2018/schedule/event/python3/ My talk is focused on the migration path. How to "port" Python 2 code to Python 3, 2to3 tool, six module, things done to make the migration simpler, etc. Victor 2018-04-26 18:25 GMT+02:00 Eric Snow <ericsnowcurrently@gmail.com>:

On Thursday, April 26, 2018, Eric Snow <ericsnowcurrently@gmail.com> wrote:
nine has a very concise, if incomplete in comparison to six and 2to3, code/namespace/functional summary: https://github.com/nandoflorestan/nine/blob/master/nine/__init__.py https://github.com/benjaminp/six/blob/master/six.py http://python-future.org/overview.html#automatic-conversion-to-py2-3-compati... python-modernize, as well as custom fixers "Cheat Sheet: Writing Python 2-3 compatible code" http://python-future.org/compatible_idioms.html https://github.com/PythonCharmers/python-future
It provides future and past packages with backports and forward ports of features from Python 3 and 2.
* alternative approaches
#!/usr/bin/env python # ~~✓pythonver: > 3.3✓~~

In my opinion, the largest failure of Python 3 is that we failed to provide a smooth and *slow* transition from Python 2 and Python 3. It can be explained by the long list of backward incompatible changes. My advice would be to restrict the number of backward incompatible changes per release, and always emit a warning (at runtime) in the previous release. For example, "async" and "await" have been marked as deprecated in Python 3.5, before becoming real keywords in Python 3.6. Then people complained that they didn't see the warning which is hidden by default, but that's another topic :-) (This issue is partially solved in Python 3.7 with Nick's PEP 565 and my -X dev option). I don't think that having Python X.Y which introduces backward incompatible changes is an issue by itself. We did it multiple times during the Python 3 cycle: my PEP 446 (non-inheritable file descriptors) and PEP 475 (retry syscalls interrupted by signals, PEP co-written with Charles-François Natali) introduced backward incompatible changes in Python 3.4 and 3.5. I am aware that they broke a few applications, but it was possible to manage these issues because each release only introduced a few backward incompatible changes. The main issue is the deprecation process. Should we provide tools to automatic conversion? Should we only document the deprecation, or also emit a warning at runtime? Should the warning be displayed by default? It's also a matter of collaboration with the Python community. For example, help major Python projects to handle these changes. Nobody wants to see pip broken by the next Python release for example. It seems like we are already working closely with pip, Cython and numpy, for example. IMHO queuing backward incompatible changes until Python 4 is a very bad idea. Start to break things early :-) But always follow the deprecation process. If my vote counts, Python 4.0 should just be the version following the previous Python 3.x release, as the Linux kernel is now doing (Linux 3 and Linux 4 are just regular release, they don't break everything.) Note: Another option is to never introduce backward incompatible changes, no? ;-) Victor

Victor Stinner wrote:
In my opinion, the largest failure of Python 3 is that we failed to provide a smooth and *slow* transition from Python 2 and Python 3.
Although for some things, such as handling of non-ascii text, it's hard to see how a smooth transition *could* have been achieved. Is it a failure if we don't succeed in doing the impossible? -- Greg

On 2018-04-28, 01:33 GMT, Greg Ewing wrote:
+1 My experience told me that by far the biggest problem in porting M2Crypto was dealiing with complete mess which was whole string/bytes confusioin in py2k. I don't see how it could be overcame gradually. Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 In those days spirits were brave, the stakes were high, men were real men, women were real women and small furry creatures from Alpha Centauri were real small furry creatures from Alpha Centauri. -- Douglas Adams

On Sat, Apr 28, 2018 at 10:36 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I don't think it's your (I'm not core developer at the time) failure. On the other hand, we should avoid many changes (e.g. bytes[index]) when doing such big change next time. -- INADA Naoki <songofacandy@gmail.com>

2018-04-28 3:33 GMT+02:00 Greg Ewing <greg.ewing@canterbury.ac.nz>:
Technically, it is easy to add an option to Python 2 to raise an exception on str+unicode and str < unicode. You can imagine the same option or a different one to really get the bytes type of Python 3 (b'abc'[0] returns 97). Such option would ease a lot to port code to Python 3, since you only have to care of bytes vs Unicode issue. You don't have to worry about the looooong list of other Python 3 changes (like the new stdlib names...). Note: Python 3 Unicode is stricter in other ways, like UTF-8 reject lone surrogates. But I don't think that it matters for most users, and technically it would also be possible to add an option to Python 2 to get the new behaviour. Moreover, Python 2 *already* has an option to switch to the new division mode: $ python2 -Q new -c 'print(1/2)' 0.5 I'm proposing that a backward incompatible change must always be prepared in the previous release. Victor

On Fri, 27 Apr 2018 14:13:32 +0200 Victor Stinner <vstinner@redhat.com> wrote:
You're comparing apples and oranges. PEP 446 and PEP 475 are only noticeable by programmers of low-level system and I/O routines (*I* haven't noticed them, for example, though I did quite a bit of network programming with Python), while the changes introduced by Python 3 affect pretty much everyone, even people who only write small simple scripts. Regards Antoine.

On Fri, May 11, 2018 at 11:15:11AM -0700, Chris Barker - NOAA Federal via Python-Dev wrote:
For what it's worth, one of the programmers at my former employer decided to port their entire code base from 2.7 to 3.x without telling anyone. He got bored waiting for management permission, so he stayed back late after work one night and ported the whole thing in his own time, it took him about four hours, and then he casually announced it over IRC the next day. Now this was a small team of coders with a relatively small code base, maybe fifty to a hundred modules or so, ranging in size from a few lines to maybe a thousand. And they did have a fair set of unit tests. (Not as many as I would like, but some.) So not all Python 3 migration stories turn into horror stories :-) -- Steve

Steven D'Aprano <steve@pearwood.info>:
So not all Python 3 migration stories turn into horror stories :-)
Peter Donis and wrote "Practical Python porting for systems programmers": http://www.catb.org/esr/faqs/practical-python-porting/ We developed and applied these techniques on src (a lightweight version-control system for single-contributor projects), reposurgeon (a tool for surgery on version-control repositories, 14KLOC), doclifter (man-page to XML-DocBook markup lifter, 8KLOC), the Python components of GPSD (9KLOC) and the Python components of NTPSec (secure network time service, 16KLOC). All this code runs under either 2 nor 3 without requiring six or any other shim library. Applying the techniques is not particularly difficult. There were no horror stories at any point. I expect to keep writing Python in this polyglot idiom until 2 is obsolete enough to fall off the radar. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a> My work is funded by the Internet Civil Engineering Institute: https://icei.org Please visit their site and donate: the civilization you save might be your own.

[esr]
All this code runs under either 2 nor 3 without requiring six or any other shim library.
We've got an application that's about 500k loc, runs under both 2 and 3. It has only one shim, a 'metaclass' decorator similar to what six provides, other than that it's all quite clean 2- and 3-wise. We long ago adopted "from __future__" as a standard part of every source file, so we have internalized the Py3 print, division and import behaviors as the norm. An occasion scan with 2to3 kept us honest about list-producing vs iterator-producing functions, and renamings and such. Our major pain point was getting extension libraries that worked with 3, notably VTK and wxPython, which weren't ported completely until last year. We had been ready to switch over completely to Py3 almost four years ago, but those major pieces were missing. We have a production ready version running under 3.6, but are going to wait for the 3.7 release before cutting off support for Python 2 altogether. Of note, we did not have any Unicode issues, as we adopted wxPython's Unicode version as soon as it was available (6-7 years ago?), and had virtually no issues then or since.

On Sat, May 12, 2018 at 8:14 AM, Skip Montanaro <skip.montanaro@gmail.com> wrote:
I have found 2to3 conversion to be remarkably easy and painless.
And the whole Unicode thing is much easier.
Another point here: between 3.0 and 3.6 (.5?) -- py3 grew a lot of minor features that made it easier to write py2/py3 compatible code. u"string", b'bytes %i' % something -- and when where the various __future__ imports made available? If these had been in place in 3.0, the whole process would have been easier :-( -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Monday, May 14, 2018, Ethan Furman <ethan@stoneleaf.us> wrote:
https://github.com/python/cpython/blame/master/Lib/__future__.py

I meant the various ones that support py2/3 compatibility — I know division predates py3, not sure about the others. But it was a rhetorical question anyway :-) -CHB https://github.com/python/cpython/blame/master/Lib/__future__.py
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

On 14 May 2018 at 12:34, Chris Barker via Python-Dev <python-dev@python.org> wrote:
The __future__ imports were already there in 2.6/3.0. The other ones weren't there initially because we didn't know which things we were tempted to add back because they were actually useful, and which ones we just thought we wanted because we were used to the way the Python 2 text model worked (or failed to work, as the case may be). (The build time source code translation step was also far less effective than we hoped it was going to be, since we completely failed to account for the problem of mapping tracebacks for converted code back to the original pre-translation code) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Apr 26, 2018 at 10:25 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
As to motivation, I suppose (since I wasn't involved yet) that it was roughly: let's bite the bullet and fix unicode with a backward-incompatible release which led to: well, we're already going to break backward compatibility so we might as well get any breakage we're planning over with now so folks only have to fix their code for this release and then to: oh, and while we're at it let's clean up a bunch of cruft that's built up over the years At least that's my impression. :) -eric

On Apr 26, 2018, at 09:28, Eric Snow <ericsnowcurrently@gmail.com> wrote:
It would certainly be an interesting document, but I suspect you’ll get a bit of the old “ask 3 lawyers and get 5 opinions” kind of response. ;) As I remember it, there was definitely a feeling like, this would be our only chance to clean up some annoying cruft, and rectify some (in hindsight) incorrect design decisions made over the years, couple with a healthy dose of “we have no idea how to do the bytes/str split in a backward compatible way". There was probably a sense that the Python community was just small enough to be able to handle such a disruptive change, but wouldn’t ever be so again. The latter is definitely true today, even if the former was overly optimistic. -Barry

On Thu, 26 Apr 2018 at 10:19 Barry Warsaw <barry@python.org> wrote:
I agree with everything Barry said. There are some lessons in hindsight of how we could have handled bytes/str, but it was more of a decision of "really long transition versus a short one" -- jokes on us for what "short" became ;) -- which we simply won't make ever again.

http://pyvideo.org/pycascades-2018/bdfl-python-3-retrospective.html link to Guido’s talk, for your convenience From: Python-Dev <python-dev-bounces+tritium-list=sdamon.com@python.org> On Behalf Of Guido van Rossum Sent: Thursday, April 26, 2018 6:12 PM To: Brett Cannon <brett@python.org> Cc: Barry Warsaw <barry@python.org>; Python-Dev <python-dev@python.org> Subject: Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3 Also see my talk at PyCascades and Victor's upcoming talk at PyCon. On Thu, Apr 26, 2018, 12:02 Brett Cannon <brett@python.org <mailto:brett@python.org> > wrote: On Thu, 26 Apr 2018 at 10:19 Barry Warsaw <barry@python.org <mailto:barry@python.org> > wrote: On Apr 26, 2018, at 09:28, Eric Snow <ericsnowcurrently@gmail.com <mailto:ericsnowcurrently@gmail.com> > wrote:
It would certainly be an interesting document, but I suspect you’ll get a bit of the old “ask 3 lawyers and get 5 opinions” kind of response. ;) As I remember it, there was definitely a feeling like, this would be our only chance to clean up some annoying cruft, and rectify some (in hindsight) incorrect design decisions made over the years, couple with a healthy dose of “we have no idea how to do the bytes/str split in a backward compatible way". There was probably a sense that the Python community was just small enough to be able to handle such a disruptive change, but wouldn’t ever be so again. The latter is definitely true today, even if the former was overly optimistic. I agree with everything Barry said. There are some lessons in hindsight of how we could have handled bytes/str, but it was more of a decision of "really long transition versus a short one" -- jokes on us for what "short" became ;) -- which we simply won't make ever again. _______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

On 04/26/2018 09:02 PM, Alex Walters wrote:
http://pyvideo.org/pycascades-2018/bdfl-python-3-retrospective.html link to Guido’s talk, for your convenience
Many thanks! -- ~Ethan~

On 27 April 2018 at 03:18, Barry Warsaw <barry@python.org> wrote:
http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an... covers some of the questions Eric is asking (mostly from my PoV, but Guido corrected my answer to the initial "Why was Python 3 made incompatible with Python 2?" question shortly after I posted the first version of it). https://www.curiousefficiency.org/posts/2014/08/python-4000.html is a more retrospective-y article that looks more at the implications for Python 4. For the "What actually happened?" info, probably the 3 main documents to look at would be PEP 3000 (the process doc), PEP 3100 (accepted changes that didn't get their own PEPs), and PEP 3099 (explicitly rejected ideas that also didn't get their own PEPs). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I gave the talk "Python 3: 10 years later" at FOSDEM and Pycon Italy and will give it again at Pycon US next month: https://fosdem.org/2018/schedule/event/python3/ My talk is focused on the migration path. How to "port" Python 2 code to Python 3, 2to3 tool, six module, things done to make the migration simpler, etc. Victor 2018-04-26 18:25 GMT+02:00 Eric Snow <ericsnowcurrently@gmail.com>:

On Thursday, April 26, 2018, Eric Snow <ericsnowcurrently@gmail.com> wrote:
nine has a very concise, if incomplete in comparison to six and 2to3, code/namespace/functional summary: https://github.com/nandoflorestan/nine/blob/master/nine/__init__.py https://github.com/benjaminp/six/blob/master/six.py http://python-future.org/overview.html#automatic-conversion-to-py2-3-compati... python-modernize, as well as custom fixers "Cheat Sheet: Writing Python 2-3 compatible code" http://python-future.org/compatible_idioms.html https://github.com/PythonCharmers/python-future
It provides future and past packages with backports and forward ports of features from Python 3 and 2.
* alternative approaches
#!/usr/bin/env python # ~~✓pythonver: > 3.3✓~~

In my opinion, the largest failure of Python 3 is that we failed to provide a smooth and *slow* transition from Python 2 and Python 3. It can be explained by the long list of backward incompatible changes. My advice would be to restrict the number of backward incompatible changes per release, and always emit a warning (at runtime) in the previous release. For example, "async" and "await" have been marked as deprecated in Python 3.5, before becoming real keywords in Python 3.6. Then people complained that they didn't see the warning which is hidden by default, but that's another topic :-) (This issue is partially solved in Python 3.7 with Nick's PEP 565 and my -X dev option). I don't think that having Python X.Y which introduces backward incompatible changes is an issue by itself. We did it multiple times during the Python 3 cycle: my PEP 446 (non-inheritable file descriptors) and PEP 475 (retry syscalls interrupted by signals, PEP co-written with Charles-François Natali) introduced backward incompatible changes in Python 3.4 and 3.5. I am aware that they broke a few applications, but it was possible to manage these issues because each release only introduced a few backward incompatible changes. The main issue is the deprecation process. Should we provide tools to automatic conversion? Should we only document the deprecation, or also emit a warning at runtime? Should the warning be displayed by default? It's also a matter of collaboration with the Python community. For example, help major Python projects to handle these changes. Nobody wants to see pip broken by the next Python release for example. It seems like we are already working closely with pip, Cython and numpy, for example. IMHO queuing backward incompatible changes until Python 4 is a very bad idea. Start to break things early :-) But always follow the deprecation process. If my vote counts, Python 4.0 should just be the version following the previous Python 3.x release, as the Linux kernel is now doing (Linux 3 and Linux 4 are just regular release, they don't break everything.) Note: Another option is to never introduce backward incompatible changes, no? ;-) Victor

Victor Stinner wrote:
In my opinion, the largest failure of Python 3 is that we failed to provide a smooth and *slow* transition from Python 2 and Python 3.
Although for some things, such as handling of non-ascii text, it's hard to see how a smooth transition *could* have been achieved. Is it a failure if we don't succeed in doing the impossible? -- Greg

On 2018-04-28, 01:33 GMT, Greg Ewing wrote:
+1 My experience told me that by far the biggest problem in porting M2Crypto was dealiing with complete mess which was whole string/bytes confusioin in py2k. I don't see how it could be overcame gradually. Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 In those days spirits were brave, the stakes were high, men were real men, women were real women and small furry creatures from Alpha Centauri were real small furry creatures from Alpha Centauri. -- Douglas Adams

On Sat, Apr 28, 2018 at 10:36 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I don't think it's your (I'm not core developer at the time) failure. On the other hand, we should avoid many changes (e.g. bytes[index]) when doing such big change next time. -- INADA Naoki <songofacandy@gmail.com>

2018-04-28 3:33 GMT+02:00 Greg Ewing <greg.ewing@canterbury.ac.nz>:
Technically, it is easy to add an option to Python 2 to raise an exception on str+unicode and str < unicode. You can imagine the same option or a different one to really get the bytes type of Python 3 (b'abc'[0] returns 97). Such option would ease a lot to port code to Python 3, since you only have to care of bytes vs Unicode issue. You don't have to worry about the looooong list of other Python 3 changes (like the new stdlib names...). Note: Python 3 Unicode is stricter in other ways, like UTF-8 reject lone surrogates. But I don't think that it matters for most users, and technically it would also be possible to add an option to Python 2 to get the new behaviour. Moreover, Python 2 *already* has an option to switch to the new division mode: $ python2 -Q new -c 'print(1/2)' 0.5 I'm proposing that a backward incompatible change must always be prepared in the previous release. Victor

On Fri, 27 Apr 2018 14:13:32 +0200 Victor Stinner <vstinner@redhat.com> wrote:
You're comparing apples and oranges. PEP 446 and PEP 475 are only noticeable by programmers of low-level system and I/O routines (*I* haven't noticed them, for example, though I did quite a bit of network programming with Python), while the changes introduced by Python 3 affect pretty much everyone, even people who only write small simple scripts. Regards Antoine.

On Fri, May 11, 2018 at 11:15:11AM -0700, Chris Barker - NOAA Federal via Python-Dev wrote:
For what it's worth, one of the programmers at my former employer decided to port their entire code base from 2.7 to 3.x without telling anyone. He got bored waiting for management permission, so he stayed back late after work one night and ported the whole thing in his own time, it took him about four hours, and then he casually announced it over IRC the next day. Now this was a small team of coders with a relatively small code base, maybe fifty to a hundred modules or so, ranging in size from a few lines to maybe a thousand. And they did have a fair set of unit tests. (Not as many as I would like, but some.) So not all Python 3 migration stories turn into horror stories :-) -- Steve

Steven D'Aprano <steve@pearwood.info>:
So not all Python 3 migration stories turn into horror stories :-)
Peter Donis and wrote "Practical Python porting for systems programmers": http://www.catb.org/esr/faqs/practical-python-porting/ We developed and applied these techniques on src (a lightweight version-control system for single-contributor projects), reposurgeon (a tool for surgery on version-control repositories, 14KLOC), doclifter (man-page to XML-DocBook markup lifter, 8KLOC), the Python components of GPSD (9KLOC) and the Python components of NTPSec (secure network time service, 16KLOC). All this code runs under either 2 nor 3 without requiring six or any other shim library. Applying the techniques is not particularly difficult. There were no horror stories at any point. I expect to keep writing Python in this polyglot idiom until 2 is obsolete enough to fall off the radar. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a> My work is funded by the Internet Civil Engineering Institute: https://icei.org Please visit their site and donate: the civilization you save might be your own.

[esr]
All this code runs under either 2 nor 3 without requiring six or any other shim library.
We've got an application that's about 500k loc, runs under both 2 and 3. It has only one shim, a 'metaclass' decorator similar to what six provides, other than that it's all quite clean 2- and 3-wise. We long ago adopted "from __future__" as a standard part of every source file, so we have internalized the Py3 print, division and import behaviors as the norm. An occasion scan with 2to3 kept us honest about list-producing vs iterator-producing functions, and renamings and such. Our major pain point was getting extension libraries that worked with 3, notably VTK and wxPython, which weren't ported completely until last year. We had been ready to switch over completely to Py3 almost four years ago, but those major pieces were missing. We have a production ready version running under 3.6, but are going to wait for the 3.7 release before cutting off support for Python 2 altogether. Of note, we did not have any Unicode issues, as we adopted wxPython's Unicode version as soon as it was available (6-7 years ago?), and had virtually no issues then or since.

On Sat, May 12, 2018 at 8:14 AM, Skip Montanaro <skip.montanaro@gmail.com> wrote:
I have found 2to3 conversion to be remarkably easy and painless.
And the whole Unicode thing is much easier.
Another point here: between 3.0 and 3.6 (.5?) -- py3 grew a lot of minor features that made it easier to write py2/py3 compatible code. u"string", b'bytes %i' % something -- and when where the various __future__ imports made available? If these had been in place in 3.0, the whole process would have been easier :-( -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Monday, May 14, 2018, Ethan Furman <ethan@stoneleaf.us> wrote:
https://github.com/python/cpython/blame/master/Lib/__future__.py

I meant the various ones that support py2/3 compatibility — I know division predates py3, not sure about the others. But it was a rhetorical question anyway :-) -CHB https://github.com/python/cpython/blame/master/Lib/__future__.py
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

On 14 May 2018 at 12:34, Chris Barker via Python-Dev <python-dev@python.org> wrote:
The __future__ imports were already there in 2.6/3.0. The other ones weren't there initially because we didn't know which things we were tempted to add back because they were actually useful, and which ones we just thought we wanted because we were used to the way the Python 2 text model worked (or failed to work, as the case may be). (The build time source code translation step was also far less effective than we hoped it was going to be, since we completely failed to account for the problem of mapping tracebacks for converted code back to the original pre-translation code) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (20)
-
Alex Walters
-
Antoine Pitrou
-
Barry Warsaw
-
Brett Cannon
-
Chris Barker
-
Chris Barker - NOAA Federal
-
Eric Fahlgren
-
Eric S. Raymond
-
Eric Snow
-
Ethan Furman
-
Greg Ewing
-
Guido van Rossum
-
INADA Naoki
-
Matěj Cepl
-
Nick Coghlan
-
Serhiy Storchaka
-
Skip Montanaro
-
Steven D'Aprano
-
Victor Stinner
-
Wes Turner