[Python-Dev] Python "2migr8"

[Steve Dower]

Apologies for the slow reply, I was travelling back from PyCon.

From Guido:
> - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-)

I didn't really expect the name to stick, but Nick and I had such a good laugh that it would have been unfair not to share it :-) (though I laughed even more at 'migr2**3')

Really, I'd expect the name to be pretty boring - "Python Straddle", "Python 2 Strict" or "Python Migration Edition" (if anyone can live with Python ME ;-) ), since the aim is to discourage general use. Users can (and should) keep using Python 2.7 while the development team is using the stricter version - or however this looks for the organisations that want to use it.

> - Are you sure this isn't better directed to python-ideas first? Most ideas have to prove their worth in that list before python-dev will give them the light of day.

It was a fifty-fifty choice between here and there. Python-dev had the more recent discussion about this issue and it was also raised at the language summit.

> - When it comes to purely syntactic issues (e.g. "except x, y:") a linter or some other separate tool can handle this well (heck, you can build it into an import hook probably :-).

True, but that isn't quite the same forcing function that interpreter errors would give. Code reviews are great until all of your potential contributors start with commit rights (and you are stuck on a VCS that has no concept of commit hooks... yes, there is also a tooling issue for the people who want this, but there also isn't a "blessed" tool for this purpose).

Eric Snow suggested that this entire idea could probably be handled as an import hook, which may mean that the whole thing could just be a PyPI package that works with 2.7. If so, great! But it still needs the blessing of python-dev, and probably at least one contributor who is also a core developer.

> - When it's about backported modules, a sumo distribution is probably the way to go; when it's about renamed stdlib modules, six (perhaps an extended version) should cover you.

Agreed. The advantage of the sumo distribution is that not every enterprise finds it easy to add more dependencies (I think this discussion happened on distutils-sig) - when it's "just part of Python and Python is already approved by our lawyers/IT guys/whoever" things are significantly easier.

> - Regarding warning about the changed dict API, I wonder how you plan to implement that if you allow passing dict object back and forth between code that has opted in to single-source and code that hasn't yet. Please think through some specific examples before responding.

I haven't thought through specific examples, so instead I'll describe my vision for how this solves the issue (it finally dawned on my during a jet-lagged dream - let's see if it still makes sense :-) ):

Migrating from 2.7 to 3.3[1] causes a lot of pain - let's call it 100% pain. You have to do 100% of the migration work before it will work.

Phase 1: Migrating from 2.7 to the straddle-edition causes a small amount of unconditional warnings and performance hit (these would be for things considered "poor form" anyway, since they are errors in 3.3), but no behaviour changes. There is some pain in fixing these, let's say 10%, but it isn't pain-pain because your program still runs. You can defer this 10% pain entirely into Phase 2 if you want.

Phase 2: Migrating from straddle-edition files without the flag to files with the flag is 70-80% of pain, distributed over a long period of time, during which your program still runs.

Phase 3: Migrating from straddle-edition where every file is flagged to 3.3 covers the last 10-20% of pain. This pain is real, since your program will not run against 3.3 until it is fixed, but it is significantly smaller than going directly from 2.7.

The last phase includes the final pass of "should this string literal be bytes?" work, but even that could be partially moved into Phase 2 by failing comparisons between byte literals and strings (apologies for the simplistic idea that is probably close to impossible to implement - the fact that nobody has a good solution to this is really the core of the entire issue...).

Basically, this idea allows the migration pain to be distributed over a longer period of time.


(1: I keep using 3.3 as my target because 3.4 has no new syntax over 3.3 - the new features in 3.4 are basically irrelevant to porting. The target version is also basically irrelevant to this discussion.)

> - But the biggest issue is of course bytes vs. text. You would have to first do a careful analysis of the *whole* problem before you can even think about proposing a solution. Too many people think there is an easy solution for this; but almost everybody is focused on only part of the problem (the part that causes them immediate pain) without realizing that other people's pain may be different.

Agreed. Maybe there is a need for different levels of migration here ("use 2.7's model", "distinct bytes type that behaves like str under 2.7", "distinct bytes type that warns when mixed with str", "use 3's model") so that there are clear steps along the path to getting it right. 

> - As far as your assertion that it would have to come from python-dev because nobody outside is going to tackle it, I think it's the opposite: the core developers would prefer not to have to deal with this, while some folks outside the inner circles will not be discouraged by our opinions (e.g. Stackless is working on "Stackless 2.8").

I agree entirely, but I don't think the first part is the opposite of the second part. The "core CPython team" should probably not be the owner of this fork, but there will certainly be some overlap between the core developers and the straddle-edition team. 

> - Regarding open source projects having a reputation of "not taking contributions", I would guess that this is usually about those "contributions" violating the most basic rules of the project (and I don't mean the coding style). I don't [corrected - Steve] want to discourage discussions with users like the company you referred to, but I think it would be much more useful if they laid out their problems for us instead of expecting they can buy acceptance for a "solution" they develop in-house. We could then hopefully have a productive dialog over many months where we iterate over possible approaches that could be acceptable both to Python and to the customer. But it will take a significant investment of time on both sides -- there is no shortcut. And it's not a particularly interesting problem (for most people) to work on -- things like designing a notation for optional type declarations are always much more fun. :-)

Agreed, but python-dev has a great mix of those who are interested in making Python awesome in the future and those who want to make Python awesome now (this is especially obvious in the packaging discussions). 


From Terry:
> This should be a separate project from pydev, even if under the PSF umbrella.

Agree entirely, but as mentioned above, I'd be extremely surprised if pydev wanted to be in a position of complete denial about the project. If that were the case, I'd have concerns about using it at all.

> A reasonable request of pydev would be for python-coded stdlib modules to be updated as much as possible, if that has not already been done. No 'apply', no 'except SomeException, e'.

I think it was addressed later, but this only really helps those who copy code from the stdlib. I would not expect the stdlib files to get the "single-source only" flag - there's no reason it shouldn't match 2.7 exactly. (Or maybe there is and I haven't thought of it, but I don't think this is the reason.)

> For PSF/CPython, this is so untrue that it looks to me like an excuse to take without giving back. This might be 'unfair and incorrect', but it is my perception.

>>     > On 4/14/2014 11:32 AM, Steve Dower wrote:
>>     [...]
>>     >> However unfair
>>     >> and incorrect it may be, there is a perception in some businesses
>>     >> that open-source projects do not want contributions from them.
>
> I took this to mean that they think any 'do not want' is peculiar to them as from a business. Perhaps I was wrong to project that on them.

No, this is exactly what I meant.

Over the years there has developed a strong sense (at least on the closed-source side) of "us" versus "them", due to silly things that have been said by both sides (Microsoft was certainly guilty of this). The bridge has been burned and CPython is doing a great job of rebuilding it, but the big/established companies still need to get used to the idea that paid and unpaid developers are not enemies and we can all contribute to these projects. Slow progress, but we're getting there :-)



From Guido:
> I really hope to have a direct conversation with some companies in this situation, but unfortunately they didn't approach me at PyCon -- they only approached Steve (perhaps because he works for a brand they recognize and trust :-).

Quoting this here because it's directly relevant to the previous points. Companies that have never had dealings with open source see talking to Guido as akin to cold-calling a different company's CEO. Getting someone lurking on python-dev is a good invitation to start with, as is attending PyCon. (These companies understand conferences, especially when they have to pay to attend. Go figure...)

 
From Eric:
> This should be doable with an import hook.  For that matter it would be pretty straight-forward to provide a script around the Python binary that installs the import hook.  All this could be bundled up and distributed through the cheeseshop.

I love this solution. If it really is doable this way then it is even better than having a fork. My name suggestion for a PyPI packages would be "PyStraddle".

It'll probably be easier to get contributors to a project like this since there'll be a smaller gap between starting work and having something that runs (yes, that includes me, despite my declaration of non-interest in my initial post). 


And on that positive note I'll end this reply.

Cheers,
Steve