On Tue, Jan 5, 2010 at 8:50 AM, Michael Foord <fuzzyman@voidspace.org.uk>wrote:

On 05/01/2010 16:10, Juan Fernando Herrera J. wrote:

How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past.

I don't know about other developers, but I certainly expected Python 3 adoption to take a few years due to libraries only gradually making the jump. I also expected there to be nervousness and pain around the migration as well.

I think in fact that libraries *are* migrating and there are lots of encouraging signs. As a community we need to do all we can to promote Python 3 adoption, but I don't think there is much reason to be worried (or complacent for that matter).

Michael is right, but doesn't answer the OP's implied question "why can't 3.x be made backwards compatible with 2.6?" Unfortunately the answer isn't easy. I wish it was "because we didn't have time to do it" (since then the solution would be simply to call for more volunteers to help out) -- but unfortunately, it comes closer to "we have no idea how to do it." Py3k was designed to be backwards incompatible, because that gives the most long-term benefit of the language improvements. (Perhaps a better way of saying this is that in the design of language improvements, feature-by-feature backwards compatibility was intentionally not a requirement, even though keeping most of the language mostly unchanged *was* a requirement.) In principle it would be possible to be more backwards compatible at the syntactic level, but that's not where the pain of porting code lies -- the major hurdles are typically he new way of thinking about Unicode (bytes vs. text strings, instead of 8-bit-strings vs. Unicode strings), and the changed semantics of dictionary keys and related APIs. Since these issues typically exist across modules (passing strings and dicts between modules is common), recognizing individual modules as "2.x" vs. "3.x" isn't possible. Note, by the way, that you won't be "stuck" on 2.6 forever -- 2.7 alpha 1 was already released. -- --Guido van Rossum (python.org/~guido)

On Tue, Jan 5, 2010 at 11:21 AM, Brian Curtin <brian.curtin@gmail.com>wrote:

On Tue, Jan 5, 2010 at 10:10, Juan Fernando Herrera J. <juanfhj@gmail.com>wrote:

...

How about a new python 3 release with (possibly partial) backwards compatibility with 2.6? I'm a big 3 fan, but I'm dismayed at the way major software hasn't been ported to it. I'm eager to use 3, but paradoxically, the 3 release makes me rather stuck with 2.6. Excuse me if this has been suggested in the past.

The proper route to take, in my opinion, is to see what 2.x libraries you are using that are not 3.x compatible, run 2to3 on them, then run their test suite, and see where you get. Submit a patch or two to the library and see what happens -- it at least gets the wheels in motion.

It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired. The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think). Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source. I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker

On Tue, Jan 5, 2010 at 21:24, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

It seems like this is a problem to be addressed, then.  Let's get the "black magic" to be better specified and documented. <http://code.google.com/p/python-incompatibility/> is an interesting start on this, but it would be better if this work could be put into 2to3 fixers as well.

Some of it is about changing the code so 2to3 doesn't have to do ugly things. For example, always use iteritems(), so that 2to3 knows that items() can be used without converting it to a list, etc. Then we do have the problems with unicode vs string vs bytes, where I can't think of a clever solution except your no-op shims, which admittedly is fugly . For me that tends to turn into testing everything until the tests run on all platforms, which I guess is close to "black magic". Don't know what to do about that. python-incompatibility is about documenting all differences, and also how to make code that runs under both 2.6 and 3.0 without 2to3. But I guess it should be extended into how to spell something that is clean in both 2.6 and 3.x.

...

The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source.

Ian: If you have examples of 2to3 updated that are not appropriate (except the x.items() -> list(x.items()) they would be appreciated.

Seriously though, I find it hard to believe that this is a big problem.  The 3.x source looks pretty similar to the 2.x source, and it's good to look at both if you're dealing with a 3.x issue.

In my experience it turned out to be less of a problem than I thought. That is to some extent because the modules I've ported has had good test coverage, and I have fixed 99.9% of the bugs by making the tests pass. Developing with Distributes 2to3 support has then been quite smooth and in most cases the separation between the 2.x source and the 3.x source hasn't been a problem. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

It's not even that easy -- libraries can't apply patches for Python 3 compatibility as they usually break Python 2 compatibility. Potentially libraries could apply patches that make a codebase 2to3 ready, but from what I've seen that's more black magic than straight forward updating, as such patches have to trick 2to3 producing the output that is desired.

I wouldn't qualify it in that way. It may be necessary occasionally to trick 2to3, but that's really a bug in 2to3 which you should report, so that trickery is then a work-around for a bug - something that you may have to do with other API, as well. The "black magic" is really more in the parts that 2to3 doesn't touch at all (because they are inherently not syntactic); these are the problem areas Guido refers to. The "black magic" then is to make the same code work unmodified for both 2.x and 3.x.

The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate

I think you misunderstand. All this is necessary, but *not* to suppress 2to3 updates. More typically, it is necessary because 2to3 leaves the code unmodified either way.

Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source.

That's an issue indeed, but one that I would hope that can be fixed by improved traceback printing. It would be good if such traceback printing could make it into 2.7. Regards, Martin

Just to clarify, the black magic I'm referring to is things like:

try: unicode_ = unicode except NameError: unicode_ = str

and some other aliases like this that are unambiguous and which 2to3 won't touch (if you write them correctly). If the porting guide noted all these tricks (of which several have been developed, and I'm only vaguely aware of a few) that would be helpful. It's not a lot of tricks, but the tricks are not obvious and 2to3 gets the translation wrong pretty often without them. For instance, when I say str in Python 2 I often means bytes, unsurprisingly, but 2to3 translates both str and unicode to str.

No, that's not what's happening. Instead, str is not translated at all (i.e. it's *not* translated to str - it just isn't touched). Translating unicode to str is always correct, AFAICT. I'm not quite sure what the magic above is supposed to achieve: in 2.x, unicode_ becomes an alias to unicode, in 3.x, 2to3 translates unicode to str, and then the block becomes try: unicode_ = str except NameError: unicode_ = str so the NameError is actually never triggered. Could it be that the magic is proposed for a mode where you *don't* use 2to3?

That *nothing* translates to bytes by default (AFAIK) means that people must either be living in a bytes-free world (which sure, lots of code does) or they are using tricks not included in 2to3 itself.

By your above definition of "translates", the function "bytes" is translated to bytes (i.e. it isn't touched by 2to3).

Since 2to3 maintains line numbers yes, it wouldn't be that bad. But then I don't currently develop any code that is "installed", I only develop code that is directly from a source code checkout, and where the checkout is put on the path. I guess I could have something that automatically builds the code on every edit, and that's not infeasible. It's just not fun.

Depends on where you draw your fun from. It is indeed fun to me, but I can see why it may not be fun to you. So you could ask me to do it for you - or you may try to use what I have already done for you, so you don't have to do it.

So long as I have to support Python 2 (which is like forever) then adding Python 3 only makes development that much more complicated and much less fun, with no concrete benefits.

I doubt it will be *much* more complicated - only a little. As for concrete benefits: there may be no benefits at this point, but in the long run, starting now will have saved you a lot of pressure in the long run, and stop users from switching away from your packages because of lack of Python 3 support. Regards, Martin

Hi Ian,

The only workable workflow I've seen people propose for maintaining a single codebase with compatibility across both 2 and 3 is to use such tricks, with aliases to suppress some 2to3 updates when they are inappropriate, so that you can run 2to3 on install and have a single canonical Python 2 source. Python 2.7 won't help much (even though it is trying) as the introduction of non-ambiguous constructions like b"" aren't compatible with previous versions of Python and so can't be used in many libraries (support at least back to Python 2.5 is the norm for most libraries, I think).

Also, running 2to3 on installation is kind of annoying, as you get source that isn't itself the canonical source, so to fix bugs you have to look at the installed source and trace it back to the bug in the original source.

I suspect a reasonable workflow might be possible with hg and maybe patch queues, but I don't feel familiar enough with those tools to map that out.

That's why I am saying that we need a Code-Repository: section in PEP-345 Metadata. Let package developers keep two branches in SCM. One for 2.x and one for 3.x Let them backport features from their 3.x development series to their 2.x code base. In exactly the same way that it is done in so many other developments today. Keeping multiple branches of code is a very common thing these days. And there are tools in the outside world (bzr,svn,mercurial) that allow us to do it very easily. Let's have that support in python; it will make life easier. David