[Python-Dev] Python "2migr8"

[Steve Dower]

Just in case there's anyone out there who isn't yet sick of discussing how to proceed with Python 2.7, I have some more inputs to contribute.

To put it up front, I'm totally against "CPython 2.8" ever becoming a real thing. Anything that comes out should be seen as a migration path, not an upgrade path. I'll also admit I'm not heavily invested in working on it myself, but I had a number of conversations during PyCon (as well as being at the language summit) that puts me in a position to share the ideas and concerns that have been raised.

The main trigger was a conversation I had with two employees of a very large bank that has about 3000 Python users (not developers - mostly financial analysts) and 16 million lines of code running on 2.7. They are keen to migrate to 3.x but cannot afford to stop work entirely while their code is updated. (There was much more to the conversation than I'm relating here - I'm keeping to the directly relevant bits.)

In describing the approach they'd like to take, they made me realise that there is definitely a place for a Python that is different but mostly compatible with 2.7, in a way that 2.7.x could not be. For the sake of having a name, I'll refer to this as "Python 2migr8" (pronounced "to migrate" :) ).

The two important components of Python 2migr8 would be the ability to disable 2.7-only features, and to do so on a module-by-module basis.

My best idea so far would be to have a magic comment (to ensure 2.7 compatibility better than a "from __future__ ...") near the top of the file that marks that file as "must straddle 2.7 and 3.3". Adding this comment causes (for example) the parser to treat "except x, y" as a syntax error in this file, forces "from __future__ import ...", hides "dict.iterkeys", undefines "basestring", etc., but only for this file. (I haven't thought through all the possibilities or implications - Eric Snow said he was going to sprint on this today/tomorrow, so he'll soon have a better idea just what can be done.)

In effect, 2migr8 would be the version that *only* supports "single-source" files. This allows large code bases to progressively migrate modules from 2.x to single-source while continuing to run against Python 2.7. As files are updated, they are marked as being single-source. When all files have this marker, it should be possible to flip the switch and run with Python 3.3 or later.

You could also think of this as enabling "-3 --warnings-as-errors" for individual modules, though since the user has already opted in to 2migr8, it isn't unreasonable to make more significant changes, like having dict.keys returning a list that warns if it is mutated. This sort of warning can only really be done by changing the interpreter - static analysis just can't catch everything - and only when users accept a potential performance hit and low probability of breakage when they move to 2migr8 (followed by a not-quite-as-low probability of breaking when they eventually move from 2migr8 to 3.x, but it's still better than guaranteed breakage).

As a fork, it would also be possible to bundle the modules that have been backported, and possibly also to disallow importing deprecated stdlib modules when 2.7 functionality is disabled. As I said, I haven't thought through all the possibilities, but the general idea is to take 2.7 and *remove* features so it becomes easier to migrate.

Where does python-dev come in? Obviously this is where a fork like this would have to start - there has been such strong and public opposition to any significant changes like this that you'd be hard pressed to find someone willing to start and promote it from outside. There is also a good opportunity to make a start and directly invite those using it to contribute the rules or warnings that they need - the 3000 Python "users" I mentioned earlier are backed by a team of true developers who are more than capable of contributing, and this would be a great opportunity to directly invite them. However unfair and incorrect it may be, there is a perception in some businesses that open-source projects do not want contributions from them. I invited more than one business to have someone join python-dev and get involved during PyCon, and I heard that others did the same - it may not be at the level of employing a core developer full time, but it's the starting point that some companies will need to be able to become comfortable with employing a core dev.

I'm not pretending to have a full plan on how this will work. I was privileged to have some private conversations during PyCon that are directly relevant, so I'm bringing it here to promote the discussion. Thanks to everyone I had a chance to chat to, and to everyone generally for a great PyCon.

Cheers,
Steve