James> 2.x seems to be a fine edition of Python, why not let it keep James> going to 2.8 and beyond? Resources would be my guess. In much the same way that people move on to newer releases of Windows, Mac OSX or Linux leaving an ever-dwindling group available to support old versions, the same will be true of Python. Over time more and more of the core developers (and end users) will switch to 3.x leaving fewer and fewer people with the time or inclination to support 2.x. I think there are probably enough functional differences between 2.6 and 2.7 that releasing 2.7 makes sense. The discussion at this point should probably be what to do when 2.7 is out the door. It makes sense to me to merge the py3k branch to trunk coincident with the 2.7/3.2 releases and creation of the release27-maint and release32-maint branches. 3.3 and future versions would then be released from trunk and there would be no further 2.x releases. Skip

On Nov 2, 2009, at 10:48 PM, ssteinerX@gmail.com wrote:

A better language, i.e. Python 3.x, will become better faster without dragging the 2.x series out any longer.

If Python 2.7 becomes the last of the 2.x series, then I personally favor back porting as many features from Python 3 as possible. I still think doing so will help people migrate to Python 3 by getting their Python 2 code base as close to Python 3 as possible without biting the ultimate bullet. E.g. for me "from __future__ import absolute_import, unicode_literals" in Python 2.6 has helped quite a bit. I also think Guido's call for feature freeze makes a lot more sense when 2.7 is the EOL. Let's give people migrating to Python 3 a nice big stable target to hit. Improving the stdlib also gives people a big carrot to move. I think it's also necessary to give third party library and application authors as much help as possible to provide Python 3 compatible software. Putting together Python tools involves so many dependencies in a fairly deep stack that even one unconverted library can cause everything above it to stall on Python 2. -Barry

On Mon, Nov 2, 2009 at 9:51 PM, ssteinerX@gmail.com wrote:

BeautifulSoup, which I use every day, is one such product.  Since the crappy old SMGL parser's gone, BeautifulSoup uses the one that's left in Python 3 and it makes BeautifulSoup completely useless for my daily work.

This sounds an area where some help might be useful. Perhaps the quickest solution would simply be to copy the old crappy "sgml" based html parser into a new version of BeautifulSoup. Though I imagine what it really needs is a "quirks mode" parser that is compatible with the HTML dialect accepted by, say, IE6. Maybe a summer of code project? -- --Guido van Rossum (python.org/~guido)

On Mon, 2 Nov 2009 at 22:06, Guido van Rossum wrote:

On Mon, Nov 2, 2009 at 9:51 PM, ssteinerX@gmail.com wrote:

...

BeautifulSoup, which I use every day, is one such product. �Since the crappy old SMGL parser's gone, BeautifulSoup uses the one that's left in Python 3 and it makes BeautifulSoup completely useless for my daily work.

This sounds an area where some help might be useful. Perhaps the quickest solution would simply be to copy the old crappy "sgml" based html parser into a new version of BeautifulSoup. Though I imagine what it really needs is a "quirks mode" parser that is compatible with the HTML dialect accepted by, say, IE6. Maybe a summer of code project?

It's not a matter of quirks. It's a matter of being able to parse truly broken html/xml, which browsers unfortunately do too well for everyone else's sanity. So, call it a "sloppy mode" parser, and then yes, that would solve the problem. --David (RDM)

On Nov 2, 2009, at 11:51 PM, ssteinerX@gmail.com wrote:

I agree as long as: A> 2.7 comes out as soon as possible, even if it's missing helpful porting features. B> 2.7 will get ONLY new features that make it easier to port to 3.x, not every feature added to 3.x or all you've done is make "Python 2.7, the Python 3 Version." and core developer time will continue to be wasted on Python 2.7 instead of moving forward.

I'm not sure I agree with that core developer time will be "wasted on Python 2.7 instead of moving forward". However, I completely understand core developers seeing Python 2.x as a dead end and not wanting to work on it. If that's a real issue, we should acknowledge that as a factor in the decision. This is a volunteer organization and if the majority of developers are sick and tired of Python 2, then it absolutely makes no sense to slog through a Python 2.7 release. I'd much rather take all the enthusiastic energy that decision will reclaim and focus it on, oh, Python 3's email package instead <wink>.

...

I also think Guido's call for feature freeze makes a lot more sense when 2.7 is the EOL. Let's give people migrating to Python 3 a nice big stable target to hit. Improving the stdlib also gives people a big carrot to move.

Agreed. And specifically NOT porting every shiny new toy from Python 3 back to 2.7 makes sure the carrots are only in the 3.x series.

Python 3's got enough carrots and sticks already. One huge carrot that will never make it back into Python 2 is bytes/strings of course. The huge stick is that Python 2 is end-of-lifed, if not now, eventually. It isn't going to get a reprieve. Everyone knows that everyone will have to get to Python 3. The question is, what can we as a community do to make that inevitability as easy to swallow as possible?

...

I think it's also necessary to give third party library and application authors as much help as possible to provide Python 3 compatible software. Putting together Python tools involves so many dependencies in a fairly deep stack that even one unconverted library can cause everything above it to stall on Python 2.

And that's one of the reasons my explorations into Python 3 have been limited to pretty much nothing.

I don't have time to do a bunch of work only to find out that the tool I absolutely have to have to finish a project doesn't have a Python 3 version or has been crippled to make a Python 3 version.

Unfortunately, I think we have to do those explorations, fail, hit roadblocks, complain, etc. but most importantly identify the packages that need to be ported. Then work with those package authors to make the upgrades happen. And improve Python and Pythonic tools so that migrations can go smoothly. Speaking as a package author, I know how much work it is just to get a bug fix release out. The three lines of code fix means 50 lines of test writing, a half a day of documenting, packaging, uploading, and announcing. Porting even one of my packages to Python 3 is a significant undertaking which frankly I don't have the cycles for. Anything large and complicated is hopeless. Witness how long and difficult it's been just to get a standard library module updated (email) and you get a sense of how much work it will be to get an entire stack of code onto Python 3.

BeautifulSoup, which I use every day, is one such product. Since the crappy old SMGL parser's gone, BeautifulSoup uses the one that's left in Python 3 and it makes BeautifulSoup completely useless for my daily work.

That's not to say I can't fix that one particular project, but customers get cranky when their project is taking longer than expected and "Oh, I'm having to convert a lot of things to use Python 3" doesn't seem to improve their mood much.

I completely agree. What happens when your application depends on a half dozen Zope packages, Twisted, and 15 or 20 other established, mature packages? It's a daunting prospect. -Barry

On Nov 3, 2009, at 1:07 AM, Martin v. Löwis wrote:

Barry Warsaw wrote:

...

On Nov 2, 2009, at 10:48 PM, ssteinerX@gmail.com wrote:

...

A better language, i.e. Python 3.x, will become better faster without dragging the 2.x series out any longer.

If Python 2.7 becomes the last of the 2.x series, then I personally favor back porting as many features from Python 3 as possible.

And if *2.6* becomes the last of the 2.x series?

Then clearly we can't back port features. I'd like to read some case studies of people who have migrated applications from 2.6 to 3.0. Having just gone through a 2 week sprint to migrate Launchpad from 2.4 to 2.6, and only making it to 2.5, I can say that I was unpleasantly surprised at the amount of work it took. A lot of that was working out the dependency upgrades, with some amount of fixing our code (mostly tests) for things that have changed (e.g. exception print/str format). We didn't make it to Python 2.6 because dealing with deprecation warnings for sha, md5, and sets (a little in our code but tons in our dependencies) consumed most of our remaining time. Given another week or so I think we would have made it to Python 2.6, but I'm not at all confident that that would have been a good enough platform to attempt an upgrade to Python 3, even if all of our very numerous large dependencies were available for Python 3. Maybe it wouldn't be so bad, but my recent experience informs me that I'm probably being too optimistic rather than too pessimistic. -Barry

...

I'd like to read some case studies of people who have migrated applications from 2.6 to 3.0.

+1, especially for packages which have a lot of C code: the current documentation is sparse :) The only helpful reference I have found so far is an email by MvL concerning psycopg2 port.

You may also want to take a look at my ZODB port: https://www.dcl.hpi.uni-potsdam.de/home/loewis/zodb/ Regards, Martin

...

You may also want to take a look at my ZODB port:

https://www.dcl.hpi.uni-potsdam.de/home/loewis/zodb/

Has that port been integrated back into the zodb project?

Only partially so.

If not, it would be interesting to know the reasons (zodb project maintainers not interested, barriers to contribution are too high, port is not 100% complete and no-one willing to take it over and complete it...?)

ZODB is a lot of layers, really (zope.interfaces, zope.lockfile, zope.proxy, ZConfig, etc..., finally ZODB). I failed to port buildout and zope.testing. With zope.testing not ported, it is not easy to actually run the test suites for all of these; this is where the integration stalled. In any case, I only did the port in September; at the DZUG sprint, people from gocept then started looking into integrating it. We didn't get through at the sprint, and I think there has been little progress since. One specific issue was about specifying the root of the zope.interfaces hierarchy. In current zope.interfaces, None could be used (implicitly) instead of Interface, to indicate the root interface type. In 3.x, this would break tests, because you can't sort a list of interfaces anymore if that list also contains None. So people decided that zope.interfaces needs to be changed to disallow None, and that was a significant change that has not yet been fully understood. Regards, Martin

On Tue, Nov 3, 2009 at 05:40, David Cournapeau wrote:

On Tue, Nov 3, 2009 at 9:55 PM, Barry Warsaw wrote:

...

Then clearly we can't back port features.

I'd like to read some case studies of people who have migrated applications from 2.6 to 3.0.

+1, especially for packages which have a lot of C code: the current documentation is sparse :) The only helpful reference I have found so far is an email by MvL concerning psycopg2 port.

There used to be a page on the wiki of case studies, but I couldn't find it. But one C-specific case study is the port of Wing's extension modules to support from Python 2.0 to 3.0: http://pythonology.blogspot.com/2009/02/making-code-run-on-python-20-through... . There is also a wiki pages explicitly about porting C code: http://wiki.python.org/moin/cporting and http://wiki.python.org/moin/Py3kExtensionModules . Those latter pages include links to the official porting docs: http://docs.python.org/3.1/howto/cporting.html . And as Georg pointed out there is always the python-porting mailing list: http://mail.python.org/mailman/listinfo/python-porting . I'm afraid there is some FUD going around here, which is understandable since no one wants to burn a ton of time on something that will be difficult or take a lot of time. But I have not heard anyone in this email thread (or anywhere for that matter) say that they tried a port in earnest and it turned out to be difficult. -Brett

2009/11/3 Brett Cannon :

I'm afraid there is some FUD going around here, which is understandable since no one wants to burn a ton of time on something that will be difficult or take a lot of time. But I have not heard anyone in this email thread (or anywhere for that matter) say that they tried a port in earnest and it turned out to be difficult.

Sadly, what I've heard a lot of is people (projects) saying that they *won't* try a port (yet, for the forseeable future, take your pick) because they *expect* it to be difficult. The worst thing is that even Martin's (and probably others') efforts in actually producing ports, and demonstrating that it's not difficult, don't seem to be changing minds. FWIW, I did a quick survey of some packages (a sampling of packages I've used or considered using in the past): Twisted - no plans yet for Python 3 wxPython - no mention of Python 3 numpy - no plans yet for Python 3 pyQt - supports Python 3 cx_Oracle - supports Python 3 pywin32 - supports Python 3 pygame - python 3 support "mostly completed" Django - not yet, FAQ suggests it will be "a year or two" TurboGears - Python 3 "currently unsupported", no timescale given PIL - 1.1.7 (due very soon) supports Python 3 lxml - supports Python 3 pyCrypto - doesn't appear to support Python 3 yet gmpy - 1.10 beta supports Python 3 pyYaml - supports Python 3 mod_wsgi - 3.0 RC 5 supports Python 3 (but see below) Here, "supports Python 3" either means that explicit support is mentioned on the website, or Windows binaries exist. I guess there's also some pure python code that "might work", possibly only requiring testing to confirm this. That's a lot better picture than I expected. How come this message isn't getting across? I suspect the big answer is that there's no web framework (that I'm aware of) which works with Python 3 - mod_wsgi supports Python 3, but from what I've seen on web-sig, the WSGI picture for Python 3 is unclear, to say the least. It seems to me that Python 3 adoption is pretty healthy, at least in these terms. If a credible Python 3 web development framework appeared (ie, one of the "big ones" got a port done) things would start to be in pretty good shape. Paul.

Antoine Pitrou writes:

Paul Moore writes:

...

TurboGears - Python 3 "currently unsupported", no timescale given

TurboGears is Pylons-based, so I suppose the actual question is when Pylons gets ported.

And there's the rub. I expect this general problem of “project FOO depends on library BAR, so there's no point thinking about migration of FOO until BAR is ready for Python 3” will prove to be quite common in widespread projects. -- \ “If you do not trust the source do not use this program.” | `\ —Microsoft Vista security dialogue | _o__) | Ben Finney

Antoine Pitrou wrote:

Paul Moore writes:

...

FWIW, I did a quick survey of some packages (a sampling of packages I've used or considered using in the past):

Twisted - no plans yet for Python 3

Well Twisted depends on zope.interface which is not ported yet.

That's not strictly true, see http://svn.zope.org/zope.interface/branches/regebro-python3/ While this isn't released yet, Lennart and myself have been working to make it work on 2.x and 3.x. So porting activities *could* start now (requiring 3.x user to use that branch). Regards, Martin

2009/11/4 Glyph Lefkowitz :

On Nov 3, 2009, at 5:16 PM, Paul Moore wrote:

...

2009/11/3 Brett Cannon :

...

I'm afraid there is some FUD going around here, which is understandable since no one wants to burn a ton of time on something that will be difficult or take a lot of time. But I have not heard anyone in this email thread (or anywhere for that matter) say that they tried a port in earnest and it turned out to be difficult.

...

FWIW, I did a quick survey of some packages (a sampling of packages I've used or considered using in the past):

Twisted - no plans yet for Python 3

Speaking of FUD, we've had a plan for Python 3 support for some time: [...]

Thanks, and my sincere apologies for spreading FUD - I did try to find details, and in fact I did spot a couple of the specific python 3 compatibility tickets you mentioned, but missed the link back to the master plan. Having said that,

 http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-t...

seems pretty negative in terms of when Twisted will support Python 3. Based on my reading, the focus is more on when 2.x support will be dropped than on when there will be a version of Twisted which works with Python 3 (which I'd expect to be much sooner!)

If you're interested in helping, our core team has all not had much time for Twisted lately and we need volunteers who are interested in doing code reviews and becoming a committer to help shepherd these tickets through the process.

Personally, I don't *use* twisted. I occasionally play with it, and I sometimes end up using applications which rely on it, but I don't use it myself. So I couldn't be much direct help. (And yes, I know that means I'm not one of your users, so me asking for Python 3 support isn't an issue. No problem there). Paul.

On Nov 3, 2009, at 10:54 AM, Guido van Rossum wrote:

Ouch. sets. I'm guessing you are referring to code that was still using the pre-2.4 sets.py module? Yes, that must have been painful.

Indeed. :) What's nice though is that these changes could be made for the Python 2.5 branch and didn't have to wait until 2.6 (e.g. sha/md5 -

hashlib, sets -> builtin-set).

What was actually most painful about this were all the tests that were checking stdout/stderr for clean subprocess runs. The DeprecationWarnings that were produced killed us for a long while because so many of those tests failed. It's actually quite difficult to suppress DeprecationWarnings across the board, especially when you have lots of subprocess that call third party code (where most of the deprecated code lived in, and which we couldn't change). We added filterwarnings where we could, and called subprocesses with -W every place we thought we needed to but we never did kill them all off. I think a $PYTHONWARNING environment variable might have helped here.

...

Given another week or so I think we would have made it to Python 2.6, but I'm not at all confident that that would have been a good enough platform to attempt an upgrade to Python 3, even if all of our very numerous large dependencies were available for Python 3. Maybe it wouldn't be so bad, but my recent experience informs me that I'm probably being too optimistic rather than too pessimistic.

There are two stages of porting to 2.6 you have to go through. The first one, which you would probably have reached in that week, is running on 2.6 period. You can then release your code for the benefit of others wanting to use it on 2.6.

Yep.

But the second stage, which can take much more time (depending on the state of your code base) is to run on 2.6 *free of warnings with the -3 flag on*. You pretty much have to consider this a separate port, and it is here where you do much of the prep work for 3.x (at least for Python code -- for C extensions it's not so clear). The good news is that you can claim 2.6 support before you've even started this stage; the other good news is that doing this right will really help you prepare for 3.x. And in most cases you can even (with some effort) maintain compatibility with 2.5 or even 2.4 -- though you may have to work around some things like the md5 and sha warnings.

The bad news is that this stage may well take more time than porting from 2.4 to 2.6.

We have another window for this (since it's all open source I don't mind talking about it) when the next version of Ubuntu comes out. Running with the -3 is a great idea, and something I will definitely try after finishing the straight 2.6 port. -Barry

On Nov 3, 2009, at 1:07 AM, Martin v. Löwis wrote:

Barry Warsaw wrote:

...

On Nov 2, 2009, at 10:48 PM, ssteinerX@gmail.com wrote:

...

A better language, i.e. Python 3.x, will become better faster without dragging the 2.x series out any longer.

If Python 2.7 becomes the last of the 2.x series, then I personally favor back porting as many features from Python 3 as possible.

And if *2.6* becomes the last of the 2.x series?

Then I'd guess that that would annoy the crap out of everyone who's put so much effort into 2.7 and all of us who have been waiting for what I hope will finally be the "now ports way more easily to 3.0" version. S

skip@pobox.com schrieb:

Martin> And if *2.6* becomes the last of the 2.x series?

With all due respect, I don't think you can make that decision now. The time to have stated 2.6 would be the end of the 2.x line was when 2.6 was released. At that point instead of opening up the trunk for changes you would have closed it off or merged the py3k branch to trunk. 2.6.0 was released over a year ago and there has been no effort to suppress bug fix or feature additions to trunk since then. If you call 2.6 "the end of 2.x" you'll have wasted a year of work on 2.7 with about a month to go before the first 2.7 alpha release.

+1. Georg

Antoine Pitrou schrieb:

Georg Brandl writes:

...

skip <at> pobox.com schrieb:

...

Martin> And if *2.6* becomes the last of the 2.x series?

With all due respect, I don't think you can make that decision now. The time to have stated 2.6 would be the end of the 2.x line was when 2.6 was released. At that point instead of opening up the trunk for changes you would have closed it off or merged the py3k branch to trunk. 2.6.0 was released over a year ago and there has been no effort to suppress bug fix or feature additions to trunk since then. If you call 2.6 "the end of 2.x" you'll have wasted a year of work on 2.7 with about a month to go before the first 2.7 alpha release.

+1.

+1 as well. Besides, it is much better communication to release 2.7 and say up front that it will be the last major release in the 2.x line. Announcing that there won't be a 2.7, however, would give the impression of poor planning.

And of a rush to get rid of 2.x. Georg

On Nov 4, 2009, at 1:06 AM, Lennart Regebro wrote:

2009/11/3 ssteinerX@gmail.com :

...

On Nov 2, 2009, at 7:26 PM, James Y Knight wrote:

...

It really sounds like you're saying that switching to 3.x isn't worth the cost to you, but you want to force people (including yourself) to do so anyways, because ...?

Because that's the future of Python

Or not. Maybe it's a dead branch of Python?

Maybe the 3.x line should just be put out of our misery, merged back to 2.7, 2.8, 2.9, and proceed as Glyph suggested in passing with increasing levels of deprecation until it just turns into 3.x on its own by running out of numbers. S

On Wed, Nov 4, 2009 at 10:39 AM, Carl Trachte wrote:

On 11/4/09, ssteinerX@gmail.com wrote:

...

Maybe the 3.x line should just be put out of our misery, merged back to 2.7, 2.8, 2.9, and proceed as Glyph suggested in passing with increasing levels of deprecation until it just turns into 3.x on its own by running out of numbers.

<delurk> As a user, I'm horrified.  Granted, I'm not the most high powered user, but . . . my employer is already providing me with a 3.0 Python version on one of my work computers with the expectation that I'll be using it more and more.

Sorry to butt in, but is this a joke?  I thought all this was hashed out prior to inventing python 3.0. </delurk>

I have no idea who "ssteinerX" is. He certainly doesn't speak for the core developers. -- --Guido van Rossum (python.org/~guido)

Carl Trachte wrote:

On 11/4/09, ssteinerX@gmail.com wrote:

...

Maybe the 3.x line should just be put out of our misery, merged back to 2.7, 2.8, 2.9, and proceed as Glyph suggested in passing with increasing levels of deprecation until it just turns into 3.x on its own by running out of numbers.

<delurk> As a user, I'm horrified. Granted, I'm not the most high powered user, but . . . my employer is already providing me with a 3.0 Python version on one of my work computers with the expectation that I'll be using it more and more.

Sorry to butt in, but is this a joke? I thought all this was hashed out prior to inventing python 3.0. </delurk>

Don't worry, 3.x is still the future of the Python language. Some of the interested onlookers are just rehashing discussions that happened years ago *before* the 3.x branch was created. It boils down to the fact that the real beneficiaries of the 2.x to 3.x transition are the people that aren't using Python yet, so existing users (especially maintainers of large libraries and frameworks) bear a disproportionate amount of the cost of the transition while gaining little of the benefit. They're understandably irritated by that and the situation is likely to take a couple more years to sort itself out. While it may be hard to tell without knowing who is and isn't a core developer, the only point seriously under discussion is whether there is going to be a 2.8 after 2.7, and the current answer to that is looking to be "probably not". Planning on that basis probably isn't a bad idea. Even if we do decide to create a 2.8 after 3k has already been merged back to the trunk, a new 2.8 development branch could easily be created based on the 2.7 maintenance branch instead of the trunk. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Thu, Nov 5, 2009 at 09:34, Yuvgoog Greenle wrote:

On Thu, Nov 5, 2009 at 2:31 PM, Nick Coghlan wrote:

...

While it may be hard to tell without knowing who is and isn't a core developer

Maybe there should be badges or something, hmmm, sounds like making an svn-commits-hall-of-fame for python could be a nice project.

You might be interested in http://www.ohloh.net/p/python/contributors . -Brett

--yuv

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

On Wed, Nov 4, 2009 at 10:20, ssteinerX@gmail.com wrote:

On Nov 4, 2009, at 1:06 AM, Lennart Regebro wrote:

...

2009/11/3 ssteinerX@gmail.com :

...

On Nov 2, 2009, at 7:26 PM, James Y Knight wrote:

...

It really sounds like you're saying that switching to 3.x isn't worth the cost to you, but you want to force people (including yourself) to do so anyways, because ...?

Because that's the future of Python

Or not. Maybe it's a dead branch of Python?

Maybe the 3.x line should just be put out of our misery, merged back to 2.7, 2.8, 2.9, and proceed as Glyph suggested in passing with increasing levels of deprecation until it just turns into 3.x on its own by running out of numbers.

I am going to say this once: we are not killing off Python 3. First off, Python 3 is not even a year old! Considering people have not fully migrated to 2.6, should we kill it off as well? There is a certain lack of perspective on time scale. This is especially true when Guido himself has said on multiple occasions that moving the community to 3.x would be a mult-year process, as in 3-5 years process, not 11 months. Second, the people calling for us to potentially kill 3.x and just keep 2.x floating along have yet to say that they have tried porting their code and that it was difficult. Every person who has stepped forward stating they have done a port has said it was actually relatively straight-forward. Not only that, we have anecdotal evidence from multiple people that you can support code way back to whatever old version of Python RHEL is running. Third, the same people calling for the death of 3.x have not suggested they have used it extensively (if at all). I have yet to hear anyone say that 3.x is not at least a nice improvement, if not a huge one. I for one find 3.x more enjoyable to work in than 2.x, and that's saying a lot since I obviously loved Python 2.x enough to get involved in its development. I have also never heard anyone ever say, "I gave 3.x a fair shake and honestly, I wish I had not wasted the time." Wait until 3to2 gets to a good state (which will happen; it's my next project -- after I either get us moved to Hg or I simply give up on it -- and I know I am not the only core developer planning on making it happen). I realize that there is some fear that it will be time wasted if people port their code to 3.x if it somehow burns out. But do you honestly think that python-dev would leave you hanging like that? Let's take a worst-case scenario here and say that direct pick-up of 3.x after a couple years never happens. Fine, we then begin to backport features. But if you already ported your code then chances are you already support the new features. And you know what one of the first things we would back port would be? Unicode strings and bytes. And since that is the hardest thing to port to, you will have not wasted any time. In other words the calling for the death of 3.x is rather premature and honestly unfair until people have actually tried to port their code in earnest and it has been a couple of years for the community to catch up to what python-dev is pushing out the door (which always takes a while no matter what minor version has been released). -Brett

On Mon, Nov 2, 2009 at 16:26, James Y Knight wrote:

On Nov 2, 2009, at 6:24 PM, ssteinerX@gmail.com wrote:

...

       +1 on 2.7 being the last of the 2.x series.  Enough already!

-1. (not that it matters)

...

I, personally, haven't even written my first line of 3.x code, nor have I had any good reason to.

Me neither.

...

If I saw the actual end of the line at 2.7, I would actually start looking for 3.x versions of my favorite tools and would be much more inclined to help push them along ASAP.

I'd probably keep using 2.7 to be able to keep using those tools, instead.

...

Right now, so much that I use on a daily basis doesn't even have a 3.x roadmap, much less any sort of working implementation, that I don't see switching to 3.x ever unless the 2.x line ends, and soon!

I don't see switching to 3.x anytime soon either. But what's the rush?

2.x seems to be a fine edition of Python, why not let it keep going to 2.8 and beyond? Then you wouldn't have to switch to 3.x at all, and that'd save you a ton of work. (and save all the people you will have to convince to make a 3.x roadmap and do the port a ton of work too!)

It really sounds like you're saying that switching to 3.x isn't worth the cost to you, but you want to force people (including yourself) to do so anyways, because ...?

... I think a decent number of us no longer want to maintain the 2.x series. Honestly, if we go past 2.7 I am simply going to stop backporting features and bug fixes. It's just too much work keeping so many branches fixed. -Brett

Sturla Molden wrote:

I'd just like to mention that the scientific community is highly dependent on NumPy. As long as NumPy is not ported to Py3k, migration is out of the question. Porting NumPy is not a trivial issue. It might take a complete rewrite of the whole C base using Cython. NumPy's ABI is not even PEP 3118 compliant. Changing the ABI for Py3k might break extension code written for NumPy using C. And scientists tend to write CPU-bound routines in languages like C and Fortran, not Python, so that is a major issue as well. If we port NumPy to Py3k, everyone using NumPy will have to port their C code to the new ABI. There are lot of people stuck with Python 2.x for this reason. It does not just affect individual scientists, but also large projects like IBM and CERN's blue brain and NASA's space telecope. So please, do not cancel 2.x support before we have ported NumPy, Matplotlib and most of their dependant extensions to Py3k.

What will it take to *start* the port? (Or is it already underway?) For many projects I fear that it is only the impending obsolescence (real rather than theoretical) of Python 2 that will convince projects to port. Python 2.X is not about to 'stop working', but there will come a point where it will 'stop being worked on'. All the best, Michael

The community of scientists and engineers using Python is growing, but shutting down 2.x support might bring an end to that.

Sturla Molden

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/

David Cournapeau writes:

To answer your question, the main issues are: - are two branches are necessary or not ? If two branches are necessary, I think we simply do not have the resources at the moment. - how to maintain a compatible C API across 2.x and 3.x - is it practically possible to support and maintain numpy from 2.4 to 3.x ? For example, I don't think the python 2.6 py3k warnings are very useful when you need to maintain compatibility with 2.4 and 2.5.

You should ask all those questions on the dedicated mailing-list: http://mail.python.org/mailman/listinfo/python-porting Regards Antoine.

On Tue, Nov 3, 2009 at 8:47 AM, ssteinerX@gmail.com wrote:

On Nov 3, 2009, at 4:55 AM, David Cournapeau wrote:

...

On Tue, Nov 3, 2009 at 6:13 PM, Michael Foord wrote:

There is also little documentation on how to port a significant C codebase to py3k.

Now there's a good Summer of Code project:  to produce a pre-processor that will flag all C constructs that need to be modified in some way, with direct pointers to the relevant documentation, and "code suggestions" wherever practicable.

S

How much interest is there in this? Geremy Condra

It's pretty easy to make Python source that works under 2.6 and 3.x. It's basically impossible to make Python source that works under 2.4/2.5 and 3.x. You may be able to write code that works under 2.4/2.5 and works cleanly with 2to3 to produce 3.x code. I haven't tried that route, though in theory it should work. All you really need is syntax compatibility.

I have tried that route for a number of projects, and I think it works really well. It is also supported by distribute.

In a nutshell, I don't think you need two branches to support an extension module on Python 2 and Python 3.

YMMV.

Exactly my experience as well. Regards, Martin

On Tue, Nov 3, 2009 at 6:14 AM, Daniel Stutzbach wrote:

It's pretty easy to make Python source that works under 2.6 and 3.x.  It's basically impossible to make Python source that works under 2.4/2.5 and 3.x.

This keeps getting quoted later in the thread so I just wanted to say again that this is not true. The argparse module supports Python 2.3-3.1 all from the same codebase, and it really wasn't all that hard: http://code.google.com/p/argparse/source/detail?r=12 Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

[Michael Foord]

What will it take to *start* the port? (Or is it already underway?) For many projects I fear that it is only the impending obsolescence (real rather than theoretical) of Python 2 that will convince projects to port.

FWIW, I do not buy into the several premises that have arisen in this thread: * For 3.x to succeed, something bad has to happen to 2.x. (which in my book translates to intentionally harming 2.x users, either through neglect or force, in order to bait them into switching to 3.x). * Core developers will are losing time supporting 2.x. (backports take some time but it is small in comparison to getting a patch to work in the first place -- if anyone can comment on this assertion, it is the people who have been doing it already (such as AP, MD, BP, GB, and myself)). * That 3.x has proven its readiness to supplant 2.x. (It hasn't been exercised that heavily and there are a lot of things that may or may not prove to be successful in the end -- bytes/text issues, tuple comparison challenges, new io, mapping views with set operations, etc). In all these matters, I think the users should get a vote. And that vote should be cast with their decision to stay with 2.x, or switch to 3.x, or try to support both. We should not muck with their rational decision making by putting "carrots" in one pile and abandoning the other. Raymond P.S. I found it curious that one of the strongest proponents of killing 2.x also mentioned that he has never written a line of 3.x code. Since this discussion is a matter of great consequence, I would hope that advocates will only take informed positions -- this isn't really time for shooting from the hip and killing 2.x.

On Nov 3, 2009, at 6:23 AM, Paul Moore wrote:

2009/11/3 Raymond Hettinger :

...

In all these matters, I think the users should get a vote. And that vote should be cast with their decision to stay with 2.x, or switch to 3.x, or try to support both. We should not muck with their rational decision making by putting "carrots" in one pile and abandoning the other.

The biggest issue to my mind is that adoption by the ultimate end users is significantly hampered by the fact that big projects like Twisted, numpy and the like, have no current plans to move to Python 3. Even end users with a reasonable level of coding expertise don't have the time or resources to offer much in the way of help with a port, when the project as a whole isn't interested in starting the process.

For what it's worth, the official position of the Twisted project is not that we have "no plan" to move to Python 3. It's that our plan is to do exactly as Raymond suggests, and give the users a vote - in this case, you vote with your patches :). We are actively and frequently encouraging our users to contribute patches that clean up warnings, which is the biggest impediment to a py3 port of Twisted. Some of you would probably expecting to whinge about how people never contribute anything, but actually, users *have* shown up and started doing this. Our biggest problem at the moment is that we don't have enough people doing code reviews so the contributions are starting to pile up. As I said in my other message, if someone would like to help, signing up to do code reviews would be a good way. Despite this progress, my hope is that there will be a robust 2.x series up through 2.9. For one thing, we have a very long row to hoe here. The migration to 3.0 is a long, tedious process with little tangible benefit. I hope that sometime in the next decade Twisted can accelerate the process of dropping old 2.x versions, but I seriously doubt we could do a feature- complete 3.1/2.6 version. I get the general impression that a 3.2/2.7 port would be more feasible; hopefully a 3.3/2.8 would be even moreso. Also, the benefits of migrating to python 3.x are still negligible, as far as I can tell. On the one hand, you've got a Python with no old- style classes and a clear unicode/bytes situation, and that's great. On the other hand, you've got NumPy, PyGTK, Unladen Swallow, PyPy, Jython, IronPython, and so on and so forth. Since I started using it, the strength of Python has been in its ecosystem, and the 3.x ecosystem is not yet viable. As long as we're tossing out modest proposals here, I still think that (as I believe James Knight already proposed) abandoning the current 3.x branch, backporting everything to 2.7, and continuing forward with a migration strategy that introduces individual deprecations every major version until 2.x == 3.x is the way to go. For example, 2.8 could emit a deprecation warning for every old-style class that was defined, 2.9 could emit a deprecation warning for every string constant declared without a 'b' or 'u' prefix unless the module in question were in "3.x mode" (i.e. no-prefix == 'u'). (I leave the determination of whether the parser should be in 3.x mode for a particular module as an exercise for the reader, but a 'from __future__' import would suffice.) I realize that there are other issues here, like the C ABI changes some NumPy folks have raised. Also, I'm not planning to actually do any *work* on this suggestion, so you can take it for what it is, which is to say, armchair quarterbacking. There have been some other comments in this thread indicating that this was not the case because some users indicated that they'd rather deal with lots of changes "all at once". My understanding is that it was done this way so that the *developers* of Python could make a clean break, and design and implement a new version of Python without being constrained by compatibility concerns. If you can show me an actual application or library developer in Python who wanted this one- big-jump migration, I will show you a crazy person. The main reason I want a long 2.x series is that I believe it would more easily allow us infrastructure folks to drop support for *older* versions. With this big 2.x->3.x chasm, I can't really see an end in sight for Twisted using Python 2.x as its _source_ language, translating with 2to3. Some projects which depend on Twisted and want new versions (and security fixes, etc) are going to want Python 2.x for a really long time. Maybe they're just really conservative, maybe they don't have a lot of maintenance energy, or maybe they have other dependencies which haven't got a port; it doesn't really matter, empirically speaking people want older versions of Python. Keep in mind also that the 2.x translation process is extremely slow and results in a clunky development process. There's no '2to3 -- interactive' commandline that lets me type python 2 at a >>> prompt and get python 3 results out so that I can try experiments on the 3.x interpreter; I have to actually put my experiments into my unit tests and wait 10 minutes to see if it works. It's like writing C++. With the 2.x series, users and operating systems seem to move on fairly rapidly, because dependencies generally continue to work if you upgrade just one version. This isn't quite as formal a requirement as I would like (warnings get generated, unit tests fail, things do break) but in practice, users can rely on it for most functionality. If 3.x could be broken into a series of transitions like that, where you can upgrade one version, fix some stuff, then upgrade another version, even if you couldn't actually support more than 2 versions at once, I think that we could pick up the migration pace to the point where we might actually be using 3.x syntax in a few years. Having a 2.x series which goes to 2.9 and then stops isn't *quite* the same thing as having one that moves over continuously to some 3.x version, but it does seem to me that by that point the chasm between versions will have narrowed to a crack, and the migration will be a little hop over it rather than the currently-required great flying leap. (To be fair, that leap is not the surface-to-orbit rocket-propelled jump that I originally expected it to be. Still, it's still not a small effort, especially if you're interfacing deeply with a lot of I/ O APIs.)

On Tue, Nov 3, 2009 at 9:51 PM, Glyph Lefkowitz wrote (amongst way too many words):

[...] For example, 2.8 could emit a deprecation warning for every old-style class that was defined, 2.9 could emit a deprecation warning for every string constant declared without a 'b' or 'u' prefix unless the module in question were in "3.x mode" (i.e. no-prefix == 'u').

This proposal is hopelessly naive. It has been considered seriously from all possible sides before, and there just isn't a way to make this work. Not even with several releases as stepping points. -- --Guido van Rossum (python.org/~guido)

For one thing, we have a very long row to hoe here. The migration to 3.0 is a long, tedious process with little tangible benefit. I hope that sometime in the next decade Twisted can accelerate the process of dropping old 2.x versions, but I seriously doubt we could do a feature-complete 3.1/2.6 version. I get the general impression that a 3.2/2.7 port would be more feasible; hopefully a 3.3/2.8 would be even moreso.

Please understand that you will not need to drop 2.x versions in order to support 3.x. Just add 3.x support now and make sure it won't break 2.x support.

Also, the benefits of migrating to python 3.x are still negligible, as far as I can tell.

For Twisted, most definitely - you will need to support 2.x and 3.x simultaneously, so you can't really benefit from 3.x-only changes for a long time to come - perhaps until a 3to2 tool has a good quality, and probably not even then (since it will restrict you what you can do in 3.x code).

On the other hand, you've got NumPy, PyGTK, Unladen Swallow, PyPy, Jython, IronPython, and so on and so forth. Since I started using it, the strength of Python has been in its ecosystem, and the 3.x ecosystem is not yet viable.

Right - the advantages wouldn't be for Twisted itself, but for users of Twisted, which would see a larger ecosystem if Twisted was available.

The main reason I want a long 2.x series is that I believe it would more easily allow us infrastructure folks to drop support for *older* versions. With this big 2.x->3.x chasm, I can't really see an end in sight for Twisted using Python 2.x as its _source_ language, translating with 2to3.

Well, 3to2 would then be an option for you: use Python 3 as the source language.

Some projects which depend on Twisted and want new versions (and security fixes, etc) are going to want Python 2.x for a really long time. Maybe they're just really conservative, maybe they don't have a lot of maintenance energy, or maybe they have other dependencies which haven't got a port; it doesn't really matter, empirically speaking people want older versions of Python.

But wouldn't these applications also break as Twisted drops support for old 2.x versions, and the applications fail to work on the newer 2.x version (say, 2.34)?

Keep in mind also that the 2.x translation process is extremely slow and results in a clunky development process. There's no '2to3 --interactive' commandline that lets me type python 2 at a >>> prompt and get python 3 results out so that I can try experiments on the 3.x interpreter; I have to actually put my experiments into my unit tests and wait 10 minutes to see if it works. It's like writing C++.

That's not my experience. I see a change in source (say, on Django) available for 3.x within 5 seconds. Regards, Martin

...

That's not my experience. I see a change in source (say, on Django) available for 3.x within 5 seconds.

This is for which version of 2to3 ? I got similar experience (several minutes), but maybe I am using 2to3 the wrong way. On my machine, with 2to3 from 3.1.1, it takes ~ 1s to convert one single file of 200 lines, and converting a tiny subset of numpy takes > one minute.

The version released with 3.1. The trick is not to recompile all 2to3 code every time you make a source change. Instead, cache the 2to3 output in the build area, and have setup.py only invoke 2to3 for the files that got modified. So whenever I make a change, I do "python3 setup.py install". This checks all timestamps, finds the modified files (which will only be one), runs 2to3 on it, and then copies it into my 3.1 installation, where I can test the change. Recompiling a single file typically takes a few seconds, or less. It would be possible to also run out of the build area; you still would need to run "setup.py build" after every change. There is already support for this in both distutils (as released in 3.1), and distribute. Regards, Martin

Mike Krell wrote:

Well, 3to2 would then be an option for you: use Python 3 as the source language.

Making life easier for 3to2 is an *excellent* rationale for backports. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Martin v. Löwis wrote:

...

Mike Krell wrote:

...

Well, 3to2 would then be an option for you: use Python 3 as the source language.

Making life easier for 3to2 is an *excellent* rationale for backports.

I'm skeptical. If new features get added to 2.7: why would that simplify 3to2? It couldn't *use* these features, since it surely would have to support 2.6 and earlier as well.

Not sure what 3to2 would do about difficult-to-convert 3.x feature (as, perhaps, the nonlocal keyword). If it currently gives up, it then may offer you to restrict your target versions to 2.7+. Not sure whether users would use that option, though - perhaps they rather stop using nonlocal in 3.x if 3to2 cannot support it for all 2.x versions they are interested in.

But surely someday 2.7 will be the oldest targetted 2.x version of Python for 3to2 and other tools (regardless of whether there's a 2.8). When that day comes, 3to2 can be made simpler, or can increase the amount of Python 3.x it can convert (or both) if we add 3.x features to 2.7. Of course, planning for a time so far in the future is difficult, and possibly pointless.

Michael Foord writes:

I would love to see nonlocal backported to 2.7 as it cleans up a simple pattern that I use relatively often for testing.

Well you know I'm sure that if someone proposes a proper patch it will eventually get accepted ;) Regards Antoine.

"Martin v. Löwis" writes:

Well, 3to2 would then be an option for you: use Python 3 as the source language.

I was under the impression that 2to3 was officially supported as part of Python, but 3to2 was a third-party tool. What's the status of 3to2 now? Is it an official part of Python? -- \ “A free society is one where it is safe to be unpopular.” | `\ —Adlai Ewing Stevenson | _o__) | Ben Finney

Ben Finney wrote:

"Martin v. Löwis" writes:

...

Well, 3to2 would then be an option for you: use Python 3 as the source language.

I was under the impression that 2to3 was officially supported as part of Python, but 3to2 was a third-party tool. What's the status of 3to2 now? Is it an official part of Python?

No, the status is exactly as you describe it. Regards, Martin

Hi Ben, On Wed, Nov 4, 2009 at 6:49 PM, Ben Finney wrote:

"Martin v. Löwis" writes:

...

Ben Finney wrote:

...

"Martin v. Löwis" writes:

...

Well, 3to2 would then be an option for you: use Python 3 as the source language.

I was under the impression that 2to3 was officially supported as part of Python, but 3to2 was a third-party tool. […] Is it an official part of Python?

No, the status is exactly as you describe it.

Okay. It's probably best for anyone with their Python developer hat on (which, in this forum, is all the time for any Python developer) to make the status of 3to2 clear when recommending it to people concerned about future plans.

Are you implying that we shouldn't recommend 3to2 to people wanting to develop in Py3k and back-translate to 2.x? Thanks, Collin Winter

Ben Finney wrote:

That 3to2 is *not* supported officially is certainly something I'd want to know if a Python core developer was recommending it to me to reassure me about the migration path to Python 3, and I didn't get that impression from the way it's been casually referenced in this discussion.

I would stick a "yet" in that sentence (i.e. "not supported officially yet"). It's certainly an addition we should be considering for 3.2 if it is sufficiently mature by then (and if not 3.2, then we can take another look for 3.3). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Glyph Lefkowitz wrote:

For what it's worth, the official position of the Twisted project is not that we have "no plan" to move to Python 3. It's that our plan is to do exactly as Raymond suggests, and give the users a vote - in this case, you vote with your patches :).

You probably will not hear from potential users who skip Twisted because it is not available for 3.x. I suspect you do not hear much either from new users who only installed 2.x to use Twisted, but would prefer 3.x. There are regular questions on python-list about 'web programming with 3.0' or some such.

For one thing, we have a very long row to hoe here. The migration to 3.0 is a long, tedious process with little tangible benefit.

One group that benefits is new Python programmers. Python 3 is easier to learn, and is being used to teach Python in at least some schools and universities, and will be used more as more libraries are available. Hardly a week goes by on Python list without someone posting a problem using 2.x that has been solved in 3.x. Another group is existing programmers who were/are sufficiently annoyed by some of the things that got cleaned up. A third group is people who want to use non-ascii in identifiers, and who are delighted now that they can. Since you do not fall in these groups, I understand your impatience and reluctance with the change. I can also imagine that Twisted may be more affected by some of the changes than most other projects. [snip more] ...

There have been some other comments in this thread indicating that this was not the case because some users indicated that they'd rather deal with lots of changes "all at once".

I wrote that based on both my reading of clp/pylist posts during the discussion of the int/int semantic change and Guido's report of private conversations he had had.

My understanding is that it was done this way so that the *developers* of Python could make a clean break, and design and implement a new version of Python without being constrained by compatibility concerns.

I do not believe that was ever intended. It certainly is not what happened; many changes were not made *because* of compatibility concerns and all went through the filter of 'is the benefit of this change worth the pain of breakage'. There is a big difference between not being straightjacketed and being unconstrained.

If you can show me an actual application or library developer in Python who wanted this one-big-jump migration, I will show you a crazy person.

Be careful of labels. Once the prolonged and intense int/int debate shifted from the ontology of ints to the pragmatics of the proposed change, most people agreed that int/int 'should' have meant float(int)/float(int) from the beginning. But some were still strongly opposed to making the change because they (understandably) did not want to have to scan (by eye) possibly 10000s or even 100000s of lines for every a/b to determine whether any fix was needed. Some said that that would be such a major change that it should not be done until there was a new major release, a Python 3 off in the then distant future. Well, that future is now. I half-jokingly suggested that the change be made on Guido's original timetable, but that the '2.5' that completed the change simply be relabeled '3.0'. I personally would have preferred that it had been completed in 2.5. But that did not happen and more changes were made once they were made, and here we are. Terry Jan Reedy

mal> I don't think users will really go for carrots. Python 2.x is mal> mature enough already and at least the users I know are really mal> happy with it (including myself). Taking that thought further back one step (or three), from http://effbot.org/tkinterbook/listbox.htm I pull this quote: In versions before Python 1.5, use string.atoi instead of int. :-) Skip

On Nov 3, 2009, at 4:58 AM, Raymond Hettinger wrote:

P.S. I found it curious that one of the strongest proponents of killing 2.x also mentioned that he has never written a line of 3.x code. Since this discussion is a matter of great consequence, I would hope that advocates will only take informed positions -- this isn't really time for shooting from the hip and killing 2.x.

Uh, that would be me. I'm only a proponent of making a decision. I *want* to have a better development language, library, and add-on tools. If 3.x is where future core development time is going to be focused, then I have faith that they will be able to make it the compelling path that it will become with Guido having put as much effort into it as he has. Or, maybe he's completely lost his mind as sometimes happens with "dictators for life" benevolent or other-wise. ;-) In any case, splitting time between 2.x and 3.x, with limited developer resources is going to lead to slower progress on both fronts. And, as you point out, if 3.x doesn't start getting the crap beat out of it in the real world sooner rather than later, we may find ourselves, collectively with a stale 2.x, an under battle-tested 3.x, and nowhere to go. S

On Tue, Nov 3, 2009 at 9:04 AM, James Y Knight wrote:

If that happens, it's not true that there's *nowhere* to go. A solution would be to discard 3.x as a failed experiment, take everything that is useful from it and port it to 2.x, and simply continue development from the last 2.x release. And from there, features can be deprecated and then removed a few releases later, as is the usual policy.

Been there, done that, on a couple other projects. It's unfortunate when you have to throw out work you've done because it failed to gain traction over the thing you tried to replace, but sometimes that's life.

I'm not ready for that yet. I think there's plenty of time before we have to agree to such a bleak view. In the mean time let's do something practical like help NumPy port to Py3k. -- --Guido van Rossum (python.org/~guido)

On Tue, Nov 3, 2009 at 9:35 AM, ssteinerX@gmail.com wrote:

On Nov 3, 2009, at 12:23 PM, Guido van Rossum wrote:

...

On Tue, Nov 3, 2009 at 9:04 AM, James Y Knight wrote:

...

If that happens, it's not true that there's *nowhere* to go. A solution would be to discard 3.x as a failed experiment, take everything that is useful from it and port it to 2.x, and simply continue development from the last 2.x release. And from there, features can be deprecated and then removed a few releases later, as is the usual policy.

Been there, done that, on a couple other projects. It's unfortunate when you have to throw out work you've done because it failed to gain traction over the thing you tried to replace, but sometimes that's life.

I'm not ready for that yet. I think there's plenty of time before we have to agree to such a bleak view. In the mean time let's do something practical like help NumPy port to Py3k.

Or, for example, Django...

Wasn't Django ported to Py3k by MvL as a demo? The problem seems to be more to port the Django *community* to Py3k... -- --Guido van Rossum (python.org/~guido)

Wasn't Django ported to Py3k by MvL as a demo? The problem seems to be more to port the Django *community* to Py3k...

Exactly so. At the last Pycon, we then agreed that I would get a branch in the Django code repository, but then progress stalled due to inactivity on boths sides. Regards, Martin

On Nov 3, 2009, at 12:43 PM, Martin v. Löwis wrote:

...

...

I'm not ready for that yet. I think there's plenty of time before we have to agree to such a bleak view. In the mean time let's do something practical like help NumPy port to Py3k.

Or, for example, Django...

See

http://wiki.python.org/moin/PortingDjangoTo3k

Well, that's certainly a start. I guess the logical question is: Now what? S

Folks: I really don't want to make anyone feel bad or to criticize, but I should mention that I have no plans to use Python 3 or to support Python 3. My best guess at this time is that the current projects that I'm involved in will still require Python 2 for the forseeable future (let's say 5 years. I can see 5 years into the future.), and that as I start new projects I will probably try out interesting alternative programming languages like Haskell, Newspeak [1], Jacaranda [2], and other new things that appear in the coming years. Of course, I reserve the right to change my mind and start using and supporting Python 3. That might happen if there is some combination of: 1. my users start asking for it (no-one has yet), 2. my dependencies start providing it (I use Python because it has Twisted. Twisted requires Python 2.), 3. it becomes more possible for me to write code which is still Python-2-compatible and also is more and more close to being Python-3-compatible. By the way, one significant detail which makes Python 3 less interesting to me is [3]. Those two languages that I mentioned -- Newspeak and Jacaranda -- both have object-capability nature. If that issue in [3] were fixed then Python 3 would join Python 2 as a language that can (with the CapPython extension) have object-capability nature. Regards, Zooko [1] http://bracha.org/Site/Newspeak.html [2] http://jacaranda.org [3] http://lackingrhoticity.blogspot.com/2008/09/cappython-unbound-methods-and-p... --- Your cloud storage provider does not need access to your data. Tahoe-LAFS -- http://allmydata.org

Arc Riley writes:

+1 on ending with 2.6.I'm the maintainer of 3rd party Python 3-only packages and have ported a few modules that we needed with some help from the 2to3 tool.  It's really not a big deal - and Py3 really is a massive improvement. The main thing holding back the community are lazy and/or obstinate package maintainers.  If they spent half the time they've put into complaining about Py3 into actually working to upgrade their code they'd be done now.

One thing you could do is explain (do you have a blog?) how Py3 is a massive improvement for you as a developer and package maintainer. We core developers obviously agree that py3k is better than 2.x, but the same opinion coming from a third-party developer would carry a different weight. Regards Antoine.

Arc Riley wrote:

+1 on ending with 2.6.

That seems precipitous.

I'm the maintainer of 3rd party Python 3-only packages and have ported a few modules that we needed with some help from the 2to3 tool. It's really not a big deal - and Py3 really is a massive improvement.

The main thing holding back the community are lazy and/or obstinate package maintainers.

I wouldn't say that. For instance, I'm just starting a refactoring that will result in getmail v.5, but I need to target Python 2.5 and up, so there's essentially no way the code will run in Python 3.x (as another list member posted). Why do I need to target Python 2.5? Because that's the most current default version of Python shipped in Debian stable and various other distributions that don't stay on the bleeding edge. getmail v.4 targeted Python 2.3 and up, getmail v.3 targeted Python 1.5.2 and up. I may be able to target Python 2.6 in a year or two, at which point Python 3 compatibility becomes a reasonable goal. Saying "2.6 is the last Python 2.x" seems to me to be a death sentence for Python 3. People will stay with 2.x much longer than you seem to want them to, and making it harder for them to upgrade will only hurt Python 3. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ -----------------------------------------------------------------------

"Martin v. Löwis" wrote:

...

I wouldn't say that. For instance, I'm just starting a refactoring that will result in getmail v.5, but I need to target Python 2.5 and up, so there's essentially no way the code will run in Python 3.x (as another list member posted).

That's a common myth. It is very well possible, using 2to3. You don't have to wait until you can drop 2.5 to start supporting 3.x, out of a single code base.

I haven't tried this, but I was relying on Daniel Stutzbach's opinion posted here earlier: http://mail.python.org/pipermail/python-dev/2009-November/093608.html It's pretty easy to make Python source that works under 2.6 and 3.x. It's basically impossible to make Python source that works under 2.4/2.5 and 3.x.

...

Why do I need to target Python 2.5? Because that's the most current default version of Python shipped in Debian stable and various other distributions that don't stay on the bleeding edge.

Are you saying that it doesn't *run* on 2.6?

No. getmail v.4 runs fine on Python 2.3.4 through 2.6.x; getmail's code has always been pretty forward-compatible.

Why? (not sure what you mean by "targetting")

By target, I mean backwards compatibility -- the minimum version of Python which is required to run getmail. getmail v.4 came out of beta about five years ago targetting Python 2.3 and higher, and 2.3 was too bleeding-edge for many users -- it wasn't shipped by many Linux distributions for a long time after getmail v.4 was released. Debian *still* shipps getmail v3 (which supports back to Python 1.5.2) today, although they also ship v4. getmail v.5 will be released in a month or three. And many of its users will still have Python 2.5, so that's what getmail has to run on. Perhaps Daniel's comment is incorrect (I have no evidence either way), but if it is true that having a single getmail codebase run on Python 2.5 and Python 3.x is "basically impossible", then I won't be too concerned about 3.x for a while yet. I've been an avid Python user and promoter since 1.2, but saying "drop Python 2.x and switch to 3 now" is simply not realistic in any of the environments in which I use Python daily. My $0.02. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ -----------------------------------------------------------------------

2009/11/3 Charles Cazabon :

"Martin v. Löwis" wrote:  It's pretty easy to make Python source that works under 2.6 and 3.x.  It's  basically impossible to make Python source that works under 2.4/2.5 and  3.x.

Without using 2to3, yes. See http://code.google.com/p/python-incompatibility/ where I've attempted to document most 2.5/3.0 incompatibilities and explain how to get around them to run the same unmodified code under 2.6 and 3.0. (It's not complete, help appreciated, especially with the C-API part)

No.  getmail v.4 runs fine on Python 2.3.4 through 2.6.x; getmail's code has always been pretty forward-compatible.

Then it's going to be dead easy to support both 2.x and 3.x by using 2to3. If you don't want to have separate packages youcan use distribute: http://packages.python.org/distribute/python3.html Also see my talk at PyCon: http://pycon.blip.tv/file/1949281/ -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

ssteinerX@gmail.com schrieb:

On Nov 3, 2009, at 12:28 PM, Arc Riley wrote:

...

The main thing holding back the community are lazy and/or obstinate package maintainers. If they spent half the time they've put into complaining about Py3 into actually working to upgrade their code they'd be done now.

That's an inflammatory, defamatory, unsubstantiated, hyperbolic, sweeping overgeneralization.

I know a few maintainers, and I have no problem seeing how Arc came to that conclusion. Georg

I'm not aware of any currently active project that isn't in the process of adding Py3 support (or who has already done so). By "most maintainers" I'm referring to the long tail; the hundreds of 3rd party modules used in niche cases and can be easily replaced by those who need the functionality they provide. Free software is an evolutionary process. Evolution dictates that those who cannot adapt will die when faced with a changing environment. It's better for us all that they do. Thus, I am not attempting to persuade anyone to upgrade their package, rather, I'm explaining why it's best we ignore them and get on with doing productive work. On Tue, Nov 3, 2009 at 4:05 PM, Mike Klaas wrote:

Be that as it may, the only way python 3 will be widely adopted if people have motivation to (need to be compatible with other libs, pressure from users, their own interest in fostering python 3.0, etc.). Deriding them as "lazy" accomplishes nothing and obscures the fact that it is the python maintainers responsibility to bring about this motivation if they want python 3.0 to be adopted. No-one is going to convert to python 3.0 because you called them lazy.

-Mike

...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com

On Tue, Nov 03, 2009 at 12:28:47PM -0500, Arc Riley wrote:

The main thing holding back the community are lazy and/or obstinate package maintainers. If they spent half the time they've put into complaining about Py3 into actually working to upgrade their code they'd be done now.

The main reason the package maintainers are so "lazy and/or obstinate" is that Python 2 is much more available than Python 3. For example, work is being completed on a python3 RPM, but it will first appear in Fedora 13, not Fedora 12. RHEL is still using Python 2.4 and won't even get Python 2.6 until RHEL 6 comes out. I don't think it's worth worrying about packages being upgraded to Python 3 when many or most users still don't have access to Python 3. In my opinion, the best place for current efforts is on infrastructure: 1) Python 3 packages for all Linux distributions (I've recently been involved with bringing Python 3 to Fedora) 2) WSGI/mod_python, etc.: according to http://wsgi.org/wsgi/Amendments_1.0 and various blog posts, the WSGI standard isn't defined for Python 3 yet, and the modwsgi page at http://code.google.com/p/modwsgi/ does not yet list a Python 3 release. 3) Numpy/scipy: as others have mentioned, numpy is essential for scientific work, and apparently the project needs help to add support for Python 3 Before basic infrastructure is available for Python 3, it's absurd to expect package maintainers to flock to it. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

Terry Reedy schrieb:

James Y Knight wrote:

...

If that happens, it's not true that there's *nowhere* to go. A solution would be to discard 3.x as a failed experiment, take everything that is useful from it and port it to 2.x, and simply continue development from the last 2.x release. And from there, features can be deprecated and then removed a few releases later, as is the usual policy.

The once 'usual policy' of removal was changed several years ago to 'defer removals until 3.0' because people wanted a more stable language and claimed that they would prefer to deal with several removals all at once. So old-style classes were kept around long past when they would have been removed under the old 'usual policy'. Ditto for old-style int / int and some others. Or one can simply recognize that 3.0 was the 'few releases later' release of that policy.

The other big change was switching to unicode strings from ascii strings with optional unicode string add-on. That was/is/will-be a hassle regardless of when and what name, but necessary for Python to be a modern world language.

From my experience, string to unicode migration really is the biggest pain when porting anything that handles real-world data.

An interesting experiment would have been to split the big changes in two parts, e.g., a 2.95 that only has the string to unicode changes, and a 3.0 that has all the rest. Of course, people would have complained about having to port twice :) Georg

Raymond Hettinger schrieb:

[Michael Foord]

...

What will it take to *start* the port? (Or is it already underway?) For many projects I fear that it is only the impending obsolescence (real rather than theoretical) of Python 2 that will convince projects to port.

FWIW, I do not buy into the several premises that have arisen in this thread:

* For 3.x to succeed, something bad has to happen to 2.x. (which in my book translates to intentionally harming 2.x users, either through neglect or force, in order to bait them into switching to 3.x).

* Core developers will are losing time supporting 2.x. (backports take some time but it is small in comparison to getting a patch to work in the first place -- if anyone can comment on this assertion, it is the people who have been doing it already (such as AP, MD, BP, GB, and myself)).

I agree. However I wouldn't want to lose the amount of work I've put into 2.7. While reviewing the 2.6 "svnmerge avail" output, I also got the impression that a *significant* number of fixes were not backported to 2.6. I don't have the time to go through a 300+ kb change log and find out what to backport, just based on commit messages that are not always clear on whether a fix or feature was added. So if we kill 2.7, we at least need to make sure no real improvements that should have been in 2.x are lost. Georg

On Tue, Nov 3, 2009 at 8:40 PM, Antoine Pitrou wrote:

Sturla Molden writes:

...

Porting NumPy is not a trivial issue. It might take a complete rewrite of the whole C base using Cython.

I don't see why they would need a rewrite.

(let me know if those numpy-specific discussions are considered OT0 There is certainly no need for a full rewrite, no. I am still unclear on the range of things to change for 3.x, but the C changes are not small, especially since numpy uses "dark" areas of python C extension. The long vs int, strings vs bytes will take some time. AFAIK, the only thing which has been attempted so far is porting our own distutils extension to python 3.x, but I have not integrated those changes yet.

between 2.x and 3.x. Cython itself is supposed to support both 2.x and 3.x, isn't it?

Yes - but no numpy code use cython ATM, except for the random generators, which would almost certainly be trivial to convert. The idea which has been discussed so far is that for *some* code which need significant changes or rewrite, using cython instead of C may be beneficial, as it would give the 3.x code "for free". Having more cython and less C could also bring more contributors - that would actually be the biggest incentive, as the number of people who know the core C code of numpy is too small.

That's interesting, because PEP 3118 was pushed mainly by a prominent member of the NumPy community and some of its features are almost dedicated to NumPy.

I have not been involved with PEP 3118 discussion, so cannot comment on the reason why it is not fully supported yet by numpy. But I think that's a different issue altogether - PEP 3118 goal is for interoperation with other packages. We can port to PEP 3118 without porting to 3.x, and we can port to 3.x without taking care of PEP 3118. David

On Nov 3, 2009, at 2:20 AM, Sturla Molden wrote:

I'd just like to mention that the scientific community is highly dependent on NumPy. As long as NumPy is not ported to Py3k, migration is out of the question. Porting NumPy is not a trivial issue. It might take a complete rewrite of the whole C base using Cython. NumPy's ABI is not even PEP 3118 compliant. Changing the ABI for Py3k might break extension code written for NumPy using C. And scientists tend to write CPU-bound routines in languages like C and Fortran, not Python, so that is a major issue as well. If we port NumPy to Py3k, everyone using NumPy will have to port their C code to the new ABI. There are lot of people stuck with Python 2.x for this reason.

It does not just affect individual scientists, but also large projects like IBM and CERN's blue brain and NASA's space telecope.

Then, perhaps, if 2.7 is known to be the last release of the 2.x line, some of those deep pockets can cough up some $$$ or developers to actually _do_ the port. A Python 3 version of NumPy might be enough of an improvement to bring *more* scientists and engineers onboard if the Python 3.x version shows what great productivity gains are to be had with Python 3.x over 2.x. S

On Tue, Nov 3, 2009 at 9:37 AM, "Martin v. Löwis" wrote:

(and no, adding things like nonlocal to 2.7 doesn't making porting of a real application or library any easier, since the existing application or library simply doesn't use that keyword. In fact, no change to 2.x can reasonably simplify porting - only changes to 3.x might - except for changes to 2to3, which can simplify porting a lot. But 2to3 should be run under 3.x, IMO.)

I agree that adding these features doesn't really help much for people that are porting from 2.x to 3.x. However, I can see it being useful for package developers who want to support both 3.x and 2.x. The more these two versions have in common the easier it will be to support them both. Obviously, this means the package might not work with 2.6 and earlier. But some users out there might be in some situation where they cannot upgrade to 3.x but can jump from 2.4/5/6 to 2.7. -Farshid

...

I would be really surprised if 2.7 would simplify porting to 3.x. How could that possibly work?

The only things I can think of that would go into this category are features like: - PEP 3118, revised buffer protocol. If the buffer API that numpy uses is not present in py3k (I'm no expert on the subject, but it seems this way from a recent thread on python-dev), then if they could move to PEP 3118 in 2.7 their migration to 3.x would be easier

But only if NumPy would drop support for 2.x, for x < 7, right? That would probably be many years in the future. Regards, Martin

On Wed, Nov 4, 2009 at 3:25 AM, "Martin v. Löwis" wrote:

But only if NumPy would drop support for 2.x, for x < 7, right? That would probably be many years in the future.

Yes. Today, given the choice of supporting py 3.x and dropping python < 2.7 and continue support for 2.4, the latter is by far my preferred choice today (RHEL still require 2.4, for example). cheers, David

On Tue, Nov 3, 2009 at 9:37 AM, "Martin v. Löwis" wrote:

(and no, adding things like nonlocal to 2.7 doesn't making porting of a real application or library any easier, since the existing application or library simply doesn't use that keyword.

Agreed.

In fact, no change to 2.x can reasonably simplify porting - only changes to 3.x might - except for changes to 2to3, which can simplify porting a lot. But 2to3 should be run under 3.x, IMO.)

Disagreed. Better "-3" warnings could make porting easier. (Not just more warnings -- "better" might mean fewer false positives for warnings already issued.) FWIW, it doesn't sound like killing 2.7 is a productive thing to do. However making 2.7 the end of the line (though with indefinite bugfix releases) might be. (Indefinite != infinite.) -- --Guido van Rossum (python.org/~guido)

Terry Reedy wrote:

Guido van Rossum wrote:

...

On Tue, Nov 3, 2009 at 9:37 AM, "Martin v. Löwis" wrote:

...

(and no, adding things like nonlocal to 2.7 doesn't making porting of a real application or library any easier, since the existing application or library simply doesn't use that keyword.

Agreed.

...

In fact, no change to 2.x can reasonably simplify porting - only changes to 3.x might - except for changes to 2to3, which can simplify porting a lot. But 2to3 should be run under 3.x, IMO.)

Disagreed. Better "-3" warnings could make porting easier. (Not just more warnings -- "better" might mean fewer false positives for warnings already issued.)

There is also Eric Smith's list to consider: PEP3118 new buffer protocol, short float repr, and maybe io.

The pure Python io module was already backported for the 2.6 release [1], as was the C API aspect of PEP 3118 [2]. Short float repr has since been backported for 2.7, as has the C accelerated io module implementation and the Python API (memoryview) aspect of PEP 3118. I believe those 3 features alone are more than enough justification to proceed with at least a 2.7 release (that is probably the point Eric was making in posting that list of features in the first place). As to how those backports can help with forward ports to Py3k, someone made the point elsewhere in the thread that testing/experimenting via 2to3 is a very C++ like development cycle - there is a long build time before you get to see the results of running a test. With features backported to 2.x, you can instead use more traditional version checks (or the interactive prompt) and get the usual quick feedback cycle via the 2.7 version, before submitting your code to the tender mercies of the 2to3 converter (or possibly avoid 2to3 altogether if the version checks turn out to suffice for a given use case). Cheers, Nick. [1] http://docs.python.org/whatsnew/2.6.html#pep-3116-new-i-o-library [2] http://docs.python.org/whatsnew/2.6.html#pep-3118-revised-buffer-protocol -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------