Mailman 3 from __past__ import division, str, etc - Python-ideas

newer
Re: [Python-ideas] RFC: bytestring...

from past import division, str, etc

older
The fools shall start sucking the...

Alejandro López Correa

8 Jan 2014 8 Jan '14

8:27 p.m.

Hi, I'm new here. I am sorry if this idea has already been discussed, but I have not found a way to search this list (I am not used to mailing lists at all). I've seen recently some discussion in reddit about python 2 vs python 3, and the slow adoption of the latter. I am proposing here pragmatic way to speed up the process of porting old code and thus solving the split in the community, that I believe it is a serious threat. It is not clean, not at all, but it might work: just give python 2 whiners what they [we] want, and do it using "from __past__ import", in a similar way "from __future__ import" is used. The advantage of this method is that porting old code would be trivial, and each module could be rewritten at its own pace (for example, when a new feature is required). The 2to3.py tool could be updated to perform as many safe changes as it could (safe in the sense of 100% certainty of not breaking anything), and import old features as needed. I am thinking both in language syntax like division behaviour, unicode, str, etc, and major library changes. Past features may be added per request, and the 2to3 tool should allow users to force the use of any of them, just in case. The whole process should be almost automatic: the user might just run the tool at the root folder of the code base, with any required command-line arguments to force some features, and the tool would generate working python3 code. There might be some issues regarding the interaction of new python3 code with code that uses old features, maintaining a more complex code base, and there might be other issues I am missing (like fundamental changes in python 3 internal architecture that can't accomodate some older features), but it might work and I think it could be useful to discuss this idea. A potential non technical problem involves users abusing this mechanism to write new code with old features, but I believe it is a minor risk if this means the whole python community finally moves to python 3. Hope this is useful. Alejandro

Attachments:

attachment.htm (text/html — 2.2 KB)

Show replies by date

Nick Coghlan

8 Jan 8 Jan

8:35 p.m.

On 9 January 2014 00:57, Alejandro López Correa wrote:

...

Hi,

I'm new here. I am sorry if this idea has already been discussed, but I have not found a way to search this list (I am not used to mailing lists at all).

I've seen recently some discussion in reddit about python 2 vs python 3, and the slow adoption of the latter. I am proposing here pragmatic way to speed up the process of porting old code and thus solving the split in the community, that I believe it is a serious threat. It is not clean, not at all, but it might work: just give python 2 whiners what they [we] want, and do it using "from __past__ import", in a similar way "from __future__ import" is used.

Hi, You may want to read through http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an... before lending too much weight to ill-informed Reddit commentary. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

9:08 p.m.

On 9 January 2014 01:05, Nick Coghlan wrote:

...

On 9 January 2014 00:57, Alejandro López Correa wrote:

...
Hi,

I'm new here. I am sorry if this idea has already been discussed, but I have not found a way to search this list (I am not used to mailing lists at all).

I've seen recently some discussion in reddit about python 2 vs python 3, and the slow adoption of the latter. I am proposing here pragmatic way to speed up the process of porting old code and thus solving the split in the community, that I believe it is a serious threat. It is not clean, not at all, but it might work: just give python 2 whiners what they [we] want, and do it using "from __past__ import", in a similar way "from __future__ import" is used.

Hi,

You may want to read through http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an... before lending too much weight to ill-informed Reddit commentary.

My apologies, that was rather rude of me when you're offering to help (I'm irritable at the moment since I've deemed it necessary to spend a bunch of time over the past week updating my Python 3 Q & A rather than enjoying my Christmas holidays, working on Python 3.4 or, this week, enjoying linux.conf.au 2014). Anyway, the problems impacting wire protocol developers are known, but it's been damnably difficult to get anything other than "I like Python 2 better" out of them when it comes to discussing possible *solutions* (although even the descriptions of the problems have been useful in guiding some changes over the course of the 3.x series). The primary pain point for developers of binary protocol manipulation code is that the Python 2 text model was *right* for boundary code that converts binary data to text or structured data. However, it's wrong for basically everything else, which is why we changed it for Python 3. The main challenge is thus getting people to stop asking the question "How do we bring back the Python 2 text model" (which is never going to happen - we changed the model for a reason), and instead ask "What changes can be made to Python 3, such as introducing additional purpose specific types, to make it a better language for wire protocol development?". There's nothing actually *saying* "thou shalt only use builtin types for manipulation of wire protocol data", but that's the way all porting efforts have been carried out to date. As part of addressing that, it's likely that certain kinds of Python 2 code will become easier to port to Python 3, but the bigger issue is to actually try to improve wire protocol development in Python 3 rather than getting stuck on just recreating the deeply flawed Python 2 model. I put some possible ideas for improvements at http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an..., but what we really need at this point is some *experimentation* with possible approaches (especially new types like asciiview and asciibytes). Regards, Nick.

...

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Alejandro López Correa

9:52 p.m.

Answering both Chris Angelico and you, I think I have not made my point clear. I am not really complaining about python 3. My opinion about the changes (at least those of which I am aware) is that they are sane. Unfortunately, many people are reluctant to change, and from what I've read that is actually a problem (not that I have actual data). I think large python 2 code bases won't change unless the benefits are larger than the costs. In the costs we have to count the available developer time, for example, and in many cases that means [a lot of] money. The idea is to offer a solution to those programmers so they can trivially port their code base, write new code in python 3 and rewrite old python 2 code as soon as possible. I am not suggesting offering back the whole python 2.7 by any means. Many changes can be safely performed by the 2to3 tool, probably. My suggestion is to offer a convenient way to bring everybody into python 3.

...

My apologies, that was rather rude of me when you're offering to help No worries.

...

The main challenge is thus getting people to stop asking the question "How do we bring back the Python 2 text model" (which is never going to happen - we changed the model for a reason), and instead ask "What changes can be made to Python 3, such as introducing additional purpose specific types, to make it a better language for wire protocol development?". There's nothing actually *saying* "thou shalt only use builtin types for manipulation of wire protocol data", but that's the way all porting efforts have been carried out to date. Enabling old functionality when required is not the same as bringing back python 2, since python 3 is there by default and python 2 code won't work by default. It means just providing a good way to make old code work. The key part is the translation from 2 to 3. This does not mean that the code has to run unchanged but that the translation may be performed automatically, at least in 99.9% of cases. In practice this could involve a mixture of changes to python 3 itself to support the 2to3 tool, and improvements to the tool.

With a 2to3 tool that covers 99.99% of the cases, we could even have .py2 modules that would be translated transparently to .py when first used, in the same way compilation works, raising an exception in case something goes wrong. Anyway, I understand it is not a clean way to proceed, but something along these lines might be the only way to speed up the adoption of python 3, and minimise the risk of defection to other languages. Cheers, Alejandro

Nick Coghlan

10:16 p.m.

On 9 January 2014 02:22, Alejandro López Correa wrote:

...

Answering both Chris Angelico and you, I think I have not made my point clear. I am not really complaining about python 3. My opinion about the changes (at least those of which I am aware) is that they are sane. Unfortunately, many people are reluctant to change, and from what I've read that is actually a problem (not that I have actual data). I think large python 2 code bases won't change unless the benefits are larger than the costs.

This is mostly a communications problem on our part. I certainly thought "5 years for Python 3 to be the default choice for new projects" was fairly straightforward to interpret, but some overly optimistic folks with less experience of corporate adoption rates managed to misinterpret that as something more like "5 years until more people are writing Python 3 code than Python 2 code". If the latter was the goal, then we'd have a crisis, but it was never the goal - Python 2 has a massive installed base, and it's going to take a long time for new Python 3 projects and Python 2 to Python 3 migrations to overtake that.

...

In the costs we have to count the available developer time, for example, and in many cases that means [a lot of] money. The idea is to offer a solution to those programmers so they can trivially port their code base, write new code in python 3 and rewrite old python 2 code as soon as possible.

I am not suggesting offering back the whole python 2.7 by any means. Many changes can be safely performed by the 2to3 tool, probably. My suggestion is to offer a convenient way to bring everybody into python 3.

This is also largely an education problem. A couple of projects have legitimate gripes about binary protocol handling in Python 3, and since they have no pressing interest in migrating (and thus little motivation to build the missing pieces of infrastructure themselves), their response has been to tell the core team "we're not migrating until *you* provide a suitable replacement for this particular Python 2 feature". It's a reasonable request, but hasn't been at the top of the core teams priority list up to this point (that's now likely to change for Python 3.5).

...

...
My apologies, that was rather rude of me when you're offering to help No worries.

...
The main challenge is thus getting people to stop asking the question "How do we bring back the Python 2 text model" (which is never going to happen - we changed the model for a reason), and instead ask "What changes can be made to Python 3, such as introducing additional purpose specific types, to make it a better language for wire protocol development?". There's nothing actually *saying* "thou shalt only use builtin types for manipulation of wire protocol data", but that's the way all porting efforts have been carried out to date. Enabling old functionality when required is not the same as bringing back python 2, since python 3 is there by default and python 2 code won't work by default. It means just providing a good way to make old code work. The key part is the translation from 2 to 3. This does not mean that the code has to run unchanged but that the translation may be performed automatically, at least in 99.9% of cases. In practice this could involve a mixture of changes to python 3 itself to support the 2to3 tool, and improvements to the tool.

There's very little actually *missing* from Python 3 now, though, and it's far from clear that the key remaining missing piece (a type for manipulating ASCII compatible binary protocol data) can't be provided as a library on PyPI.

...

With a 2to3 tool that covers 99.99% of the cases, we could even have .py2 modules that would be translated transparently to .py when first used, in the same way compilation works, raising an exception in case something goes wrong.

I'm pretty sure someone already wrote one of those - they're a problem, because they mean the tracebacks for runtime exceptions don't match the source code (that's one of the major reasons single-source approaches came to dominate as the preferred migration mechanism for libraries and frameworks, leaving 2to3 as an option mainly considered by applications that can abandon Python 2 support when migrating to Python 3).

...

Anyway, I understand it is not a clean way to proceed, but something along these lines might be the only way to speed up the adoption of python 3, and minimise the risk of defection to other languages.

We're largely happy with the rate of adoption though - there were just some folks that didn't grasp the kinds of time scales we're talking about for a migration of this magnitude. See this for more details: http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Steven D'Aprano

9 Jan 9 Jan

3:31 a.m.

On Wed, Jan 08, 2014 at 05:22:02PM +0100, Alejandro López Correa wrote: [...]

...

Anyway, I understand it is not a clean way to proceed, but something along these lines might be the only way to speed up the adoption of python 3

One assumption in this discussion, and the various related discussions on Reddit and other places, is that adoption of Python 3 is too slow and needs to be sped up. I don't believe this is true. I believe adoption is just right and exactly what should be expected. Alex Gaynor wrote a blog post a week or so ago claiming that, five years since Python 3 was first released, everyone should have migrated by now and that since only "five percent" (a figure which I believe he pulled out of thin air) have migrated, Python 3 has been a failure. I challenge that belief. I've been hanging around here and on the Python-Dev list for a long time, and while I can't find any official pronouncement, the sense has always been that Python 3 adoption will take ten years, not five. (That's my recollection -- if any of the core developers wish to correct me, please do.) Rates of adoption are much, much higher than gossip on the Internet suggests. About 70% of the top 200 projects on PyPI support Python 3, and downloads of Python 3 are very healthy, possibly even higher than downloads of Python 2. On the tutor list, I see a significant number of beginners using Python 3. It seems to me that given the circumstances, Python 3 adoption is right where we should expect it to be half-way through a decade-long process. There will be a period at the start when hardly anyone will migrate, then a period of accelerating migration, which will accelerate further when the mainstream Linux distros start shipping Python 3 as their system Python (ArchLinux is not mainstream, but Fedora is planning the change), followed by a sudden rush in another four or five years when people realise that Python 2.7 becoming unmaintained is no longer a distant prospect but is about to happen. For many people, waiting until the last minute is the most sensible thing that they can do. This gives time for the early adoptors to discover and iron out all the wrinkles and difficulties. Rather than approaching this as "Python 3 has been a failure, what can we do to save it?" we should be approaching this as "Python 3 has been a success, what lessons can we take from the early adoptors to make it even easier for the next wave of adoptors?" "from __past__ import spam" does not make it easier to adopt. It just makes it easier to *put off adopting*.

...

and minimise the risk of defection to other languages.

People threaten that, but it is an irrational threat. (Mind you, people do silly, irrational things every day.) If you think its hard to migrate from Python 2 to 3, when you get to keep 90% of your code base and most of the backward-incompatible changes are a few libraries that have been renamed and a handful of syntax changes, how hard will it be to throw away 100% of your code and start again with a completely different language? -- Steven

Alejandro López Correa

5:04 a.m.

2014/1/8 Steven D'Aprano :

...

About 70% of the top 200 projects on PyPI support Python 3, and downloads of Python 3 are very healthy, possibly even higher than downloads of Python 2.

I do not think that one is a particularly good metric. For each project hosted at PyPI how many are not there? People have personal projects, companies have internal software, and there are products that contain at least some python and are targeted at final customers, like games or Maya. Not everything is open source, but even if it is proprietary software it is good to have it since that way more jobs are offered and more people can earn money with this language, and that is a guarantee for its long-term success.

...

...
and minimise the risk of defection to other languages.

People threaten that, but it is an irrational threat. (Mind you, people do silly, irrational things every day.) If you think its hard to migrate from Python 2 to 3, when you get to keep 90% of your code base and most of the backward-incompatible changes are a few libraries that have been renamed and a handful of syntax changes, how hard will it be to throw away 100% of your code and start again with a completely different language?

I think human psychology works like that. Many people may delay the acquisition of a new car, but once they are committed to buy a new one they want the best they can afford (within their budget). Some languages may gain momentum and gain the "cool" vibe. We saw the rise of Ruby a while ago, and maybe a language that handles well multiple cores could be a strong temptation in the future. If people keep investing in python, small bits at a time, keeping their codebase always up to date, it is more difficult, IMHO, to commit to a full rewrite. Alejandro

Chris Angelico

5:24 a.m.

On Thu, Jan 9, 2014 at 10:34 AM, Alejandro López Correa wrote:

...

2014/1/8 Steven D'Aprano :

...
About 70% of the top 200 projects on PyPI support Python 3, and downloads of Python 3 are very healthy, possibly even higher than downloads of Python 2.

I do not think that one is a particularly good metric. For each project hosted at PyPI how many are not there? People have personal projects, companies have internal software, and there are products that contain at least some python and are targeted at final customers, like games or Maya.

But what IS a good metric? How are you going to measure any of that? It's better to at least use PyPI stats than to pull numbers out of a hat.

...

...
...
and minimise the risk of defection to other languages.

People threaten that, but it is an irrational threat. (Mind you, people do silly, irrational things every day.) If you think its hard to migrate from Python 2 to 3, when you get to keep 90% of your code base and most of the backward-incompatible changes are a few libraries that have been renamed and a handful of syntax changes, how hard will it be to throw away 100% of your code and start again with a completely different language?

I think human psychology works like that. Many people may delay the acquisition of a new car, but once they are committed to buy a new one they want the best they can afford (within their budget). Some languages may gain momentum and gain the "cool" vibe. We saw the rise of Ruby a while ago, and maybe a language that handles well multiple cores could be a strong temptation in the future. If people keep investing in python, small bits at a time, keeping their codebase always up to date, it is more difficult, IMHO, to commit to a full rewrite.

Maybe. But how much temptation would it need to be to induce a complete rewrite? (Mind you, it's not always a *complete* rewrite. I've been "porting" code from Win32 C++ to GTK Pike, and in the process usually shortened it by 50% or better, but mostly what I'm doing is reading the old code, taking maybe a few bits of it that are so simple they'd be the same in nearly any language, and reimplementing the original logic.) The expanded gap between Python 2.7 and Python 3.7 is mainly going to be features of 3.7 that you could choose to use now that you've ported, rather than mandatory changes. Python doesn't arbitrarily drop features or break stuff in minor releases. That means the gap between 2.7 and 3.7 will still be far FAR narrower than the gap between Python and Ruby - so, correspondingly, the temptation to switch to Ruby would have to be really strong. In the porting case I mentioned a moment ago, there really was a very strong temptation (using Win32 APIs meant I was bound to Windows (though Wine is a wonderful thing), and the C++ code was going through stupid levels of overhead to manage memory and such), so it was worth switching. I was NOT able to convince my boss to switch our web site from PHP into Python, because he just couldn't see enough benefit from changing language - but moving to a new PHP was a much lower hump to get over. (Only a few things needed changing.) ChrisA

Alejandro López Correa

5:48 a.m.

2014/1/9 Chris Angelico :

...

On Thu, Jan 9, 2014 at 10:34 AM, Alejandro López Correa wrote:

...
2014/1/8 Steven D'Aprano :

But what IS a good metric? How are you going to measure any of that? It's better to at least use PyPI stats than to pull numbers out of a hat.

The problem I see is that metric might be equal or worse than just guessing because it is clearly biased: it focuses on open source projects hosted on PyPI. It is easy to measure it, but maybe it is not good to do so if that measure is used to make important decisions. In my [very limited] experience, the number of open source projects pales in comparison to that of projects kept "in the shadows".

...

Maybe. But how much temptation would it need to be to induce a complete rewrite? (Mind you, it's not always a *complete* rewrite. I've been "porting" code from Win32 C++ to GTK Pike, and in the process usually shortened it by 50% or better, but mostly what I'm doing is reading the old code, taking maybe a few bits of it that are so simple they'd be the same in nearly any language, and reimplementing the original logic.) The expanded gap between Python 2.7 and Python 3.7 is mainly going to be features of 3.7 that you could choose to use now that you've ported, rather than mandatory changes. Python doesn't arbitrarily drop features or break stuff in minor releases. That means the gap between 2.7 and 3.7 will still be far FAR narrower than the gap between Python and Ruby - so, correspondingly, the temptation to switch to Ruby would have to be really strong. In the porting case I mentioned a moment ago, there really was a very strong temptation (using Win32 APIs meant I was bound to Windows (though Wine is a wonderful thing), and the C++ code was going through stupid levels of overhead to manage memory and such), so it was worth switching. I was NOT able to convince my boss to switch our web site from PHP into Python, because he just couldn't see enough benefit from changing language - but moving to a new PHP was a much lower hump to get over. (Only a few things needed changing.)

Fair enough. I think it is a good argument. Alejandro

Amber Yust

6:49 a.m.

Also note that even if publicly visible projects are outnumbered by private projects, the public projects tend to have a much larger impact on the overall ecosystem, because they're used by many entities (whereas private projects are typically only used by a single entity given their nature). On Jan 8, 2014 5:13 PM, "Alejandro López Correa" wrote:

...

2014/1/9 Chris Angelico :

...
On Thu, Jan 9, 2014 at 10:34 AM, Alejandro López Correa wrote:

...
2014/1/8 Steven D'Aprano :

But what IS a good metric? How are you going to measure any of that? It's better to at least use PyPI stats than to pull numbers out of a hat.

The problem I see is that metric might be equal or worse than just guessing because it is clearly biased: it focuses on open source projects hosted on PyPI. It is easy to measure it, but maybe it is not good to do so if that measure is used to make important decisions. In my [very limited] experience, the number of open source projects pales in comparison to that of projects kept "in the shadows".

...
Maybe. But how much temptation would it need to be to induce a complete rewrite? (Mind you, it's not always a *complete* rewrite. I've been "porting" code from Win32 C++ to GTK Pike, and in the process usually shortened it by 50% or better, but mostly what I'm doing is reading the old code, taking maybe a few bits of it that are so simple they'd be the same in nearly any language, and reimplementing the original logic.) The expanded gap between Python 2.7 and Python 3.7 is mainly going to be features of 3.7 that you could choose to use now that you've ported, rather than mandatory changes. Python doesn't arbitrarily drop features or break stuff in minor releases. That means the gap between 2.7 and 3.7 will still be far FAR narrower than the gap between Python and Ruby - so, correspondingly, the temptation to switch to Ruby would have to be really strong. In the porting case I mentioned a moment ago, there really was a very strong temptation (using Win32 APIs meant I was bound to Windows (though Wine is a wonderful thing), and the C++ code was going through stupid levels of overhead to manage memory and such), so it was worth switching. I was NOT able to convince my boss to switch our web site from PHP into Python, because he just couldn't see enough benefit from changing language - but moving to a new PHP was a much lower hump to get over. (Only a few things needed changing.)

Fair enough. I think it is a good argument.

Alejandro _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Nick Coghlan

4:46 p.m.

On 9 Jan 2014 09:49, "Amber Yust" wrote:

...

Also note that even if publicly visible projects are outnumbered by

private projects, the public projects tend to have a much larger impact on the overall ecosystem, because they're used by many entities (whereas private projects are typically only used by a single entity given their nature). It also mistakenly assumes our goal is to get existing *applications* to migrate. It really isn't - we're obviously delighted if app developers choose to switch (as it indicates we have created a compelling platform), but we *needed* key library and framework developers to add Python 3 support in order to bootstrap the Python 3 development ecosystem. Cheers, Nick.

...

On Jan 8, 2014 5:13 PM, "Alejandro López Correa" wrote:

...
2014/1/9 Chris Angelico :

...
On Thu, Jan 9, 2014 at 10:34 AM, Alejandro López Correa

wrote:

...

...
...
...
2014/1/8 Steven D'Aprano :

But what IS a good metric? How are you going to measure any of that? It's better to at least use PyPI stats than to pull numbers out of a hat.

The problem I see is that metric might be equal or worse than just guessing because it is clearly biased: it focuses on open source projects hosted on PyPI. It is easy to measure it, but maybe it is not good to do so if that measure is used to make important decisions. In my [very limited] experience, the number of open source projects pales in comparison to that of projects kept "in the shadows".

...
Maybe. But how much temptation would it need to be to induce a complete rewrite? (Mind you, it's not always a *complete* rewrite. I've been "porting" code from Win32 C++ to GTK Pike, and in the process usually shortened it by 50% or better, but mostly what I'm doing is reading the old code, taking maybe a few bits of it that are so simple they'd be the same in nearly any language, and reimplementing the original logic.) The expanded gap between Python 2.7 and Python 3.7 is mainly going to be features of 3.7 that you could choose to use now that you've ported, rather than mandatory changes. Python doesn't arbitrarily drop features or break stuff in minor releases. That means the gap between 2.7 and 3.7 will still be far FAR narrower than the gap between Python and Ruby - so, correspondingly, the temptation to switch to Ruby would have to be really strong. In the porting case I mentioned a moment ago, there really was a very strong temptation (using Win32 APIs meant I was bound to Windows (though Wine is a wonderful thing), and the C++ code was going through stupid levels of overhead to manage memory and such), so it was worth switching. I was NOT able to convince my boss to switch our web site from PHP into Python, because he just couldn't see enough benefit from changing language - but moving to a new PHP was a much lower hump to get over. (Only a few things needed changing.)

Fair enough. I think it is a good argument.

Alejandro _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Giampaolo Rodola'

7:33 p.m.

On Thu, Jan 9, 2014 at 12:16 PM, Nick Coghlan wrote:

...

On 9 Jan 2014 09:49, "Amber Yust" wrote:

...
Also note that even if publicly visible projects are outnumbered by

private projects, the public projects tend to have a much larger impact on the overall ecosystem, because they're used by many entities (whereas private projects are typically only used by a single entity given their nature).

It also mistakenly assumes our goal is to get existing *applications* to migrate. It really isn't - we're obviously delighted if app developers choose to switch (as it indicates we have created a compelling platform), but we *needed* key library and framework developers to add Python 3 support in order to bootstrap the Python 3 development ecosystem.

...

From a user standpoint this sort of appears as a signal which translates into "if neither big project X has migrated after 5 years why should I?". That's likely to apply even if project X is not within the list of your dependencies, because you may not depend from X now but maybe you will in

True. I think one of the key points here is that different important libs haven't been ported yet: https://python3wos.appspot.com/ Too many of them are still marked red and IMO that is the main reason why a lot of people are being so hesitant, not unicode. "boto" alone counts as hundreds of thousands potential users which simply cannot migrate. Django made the transition only a couple of months ago, which basically means it's still in a beta state, and AFAIK fundamental projects such as Twisted don't even have an ETA. Considering 5 years have passed since Python 3.0 first made it's appearance I consider this a *serious* delay. the future, either because you need X or because Y requires X in order to work. It is *crucial* for people maintaining those libraries to put Python 3 porting on top of their TODO list at the cost of not working on new features. --- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/

Nick Coghlan

10:20 p.m.

On 9 Jan 2014 22:03, "Giampaolo Rodola'" wrote:

...

On Thu, Jan 9, 2014 at 12:16 PM, Nick Coghlan wrote:

...
On 9 Jan 2014 09:49, "Amber Yust" wrote:

...
Also note that even if publicly visible projects are outnumbered by

...

...
It also mistakenly assumes our goal is to get existing *applications* to

migrate. It really isn't - we're obviously delighted if app developers choose to switch (as it indicates we have created a compelling platform), but we *needed* key library and framework developers to add Python 3 support in order to bootstrap the Python 3 development ecosystem.

True. I think one of the key points here is that different important libs haven't been ported yet: https://python3wos.appspot.com/ Too many of them are still marked red and IMO that is the main reason why a lot of people are being so hesitant, not unicode. "boto" alone counts as hundreds of thousands potential users which simply cannot migrate. Django made the transition only a couple of months ago, which basically means it's still in a beta state, and AFAIK fundamental projects such as Twisted don't even have an ETA. Considering 5 years have passed since Python 3.0 first made it's appearance I consider this a *serious* delay. From a user standpoint this sort of appears as a signal which translates into "if neither big project X has migrated after 5 years why should I?". That's likely to apply even if project X is not within the list of your dependencies, because you may not depend from X now but maybe you will in

private projects, the public projects tend to have a much larger impact on the overall ecosystem, because they're used by many entities (whereas private projects are typically only used by a single entity given their nature). the future, either because you need X or because Y requires X in order to work. It is *crucial* for people maintaining those libraries to put Python 3 porting on top of their TODO list at the cost of not working on new features. This is still focusing on migrating *existing* applications. We're not especially worried if existing applications keep using 2.7 - it's a good language that is almost certain to be commercially supported for at least another decade, even though upstream support will switch to security fix only mode in 2015. If it ain't broke (and for existing applications, 2.7 generally ain't broke), don't fix it. But if a project has persistent problems with application developers persistently introducing bugs by using 8-bit strings where they should be using Unicode, or otherwise running into the assorted bug magnets we removed in Python 3, the migration may be worth considering. A user that starts with Python 3 simply wouldn't consider a dependency like boto as an option, and would reach for asyncio rather than Twisted for their explicit asynchronous programming needs. Cheers, Nick.

...

--- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/

Andrew Barnert

11:30 p.m.

On Jan 9, 2014, at 8:50, Nick Coghlan wrote:

...

But if a project has persistent problems with application developers persistently introducing bugs by using 8-bit strings where they should be using Unicode, or otherwise running into the assorted bug magnets we removed in Python 3, the migration may be worth considering.

One thing to note: For many applications, it's not that hard to migrate to the six-able subset of 2.7/3.3. This allows 2.x-centric contributors (including those who want to be able to just use the python that Apple or Ubuntu pre-installed on their dev box), allows you to continue using py2exe for your Windows binaries, and gives you an out if you run into the "but what it we later need some library that hasn't been ported yet" problem (which I think is drastically overblown, but it's certainly a common enough fear). And meanwhile, from my experience, it's at least as hard to introduce subtle Unicode bugs in a dual-version code base as a 3.x-only code base, and just as easy to debug them, so you get at least one advantage over 2.x-only. And being able to migrate gradually instead of having a flag-day release is always nice. In my day job, I work on a project that's written in multiple languages, and the python parts are all 2.7+/3.3+. While I miss being able to use some 3.3 features, and it's annoying to deal with problems like 2.7 using too much memory when processing giant XML files or the old version of sqlite in 2.7.2 panicking over a simple union and ignoring an index, it's still far better than having to debug mojibake in 2.7, or writing in node.js or ObjC.

Giampaolo Rodola'

11:35 p.m.

On Thu, Jan 9, 2014 at 5:50 PM, Nick Coghlan wrote:

...

On 9 Jan 2014 22:03, "Giampaolo Rodola'" wrote:

...
On Thu, Jan 9, 2014 at 12:16 PM, Nick Coghlan

...
...
On 9 Jan 2014 09:49, "Amber Yust" wrote:

...
Also note that even if publicly visible projects are outnumbered by

...
...
It also mistakenly assumes our goal is to get existing *applications*

to migrate. It really isn't - we're obviously delighted if app developers choose to switch (as it indicates we have created a compelling platform), but we *needed* key library and framework developers to add Python 3 support in order to bootstrap the Python 3 development ecosystem.

True. I think one of the key points here is that different important libs haven't been ported yet: https://python3wos.appspot.com/ Too many of them are still marked red and IMO that is the main reason why a lot of people are being so hesitant, not unicode. "boto" alone counts as hundreds of thousands potential users which simply cannot migrate. Django made the transition only a couple of months ago, which basically means it's still in a beta state, and AFAIK fundamental projects such as Twisted don't even have an ETA. Considering 5 years have passed since Python 3.0 first made it's appearance I consider this a *serious* delay. From a user standpoint this sort of appears as a signal which translates into "if neither big project X has migrated after 5 years why should I?". That's likely to apply even if project X is not within the list of your dependencies, because you may not depend from X now but maybe you will in

wrote: private projects, the public projects tend to have a much larger impact on the overall ecosystem, because they're used by many entities (whereas private projects are typically only used by a single entity given their nature). the future, either because you need X or because Y requires X in order to work. It is *crucial* for people maintaining those libraries to put Python 3 porting on top of their TODO list at the cost of not working on new features.

This is still focusing on migrating *existing* applications.

I was talking about existing third party libraries (Twisted, gevent, lxml etc), not user applications. In order to port user applications you need those libraries to be ported first, and it is crucial that at least the most used ones are ported.

...

A user that starts with Python 3 simply wouldn't consider a dependency like boto as an option

Why not? Note that I picked "boto" just because it's the first in that list.

...

and would reach for asyncio rather than Twisted for their explicit asynchronous programming needs.

I would't be so sure about that. We're talking about two very mature and established projects, with tons of third-party components (see https://github.com/facebook/tornado/wiki/Links), each solving a common set of problems in their own way, which will likely continue to be used independently from asyncio (which is still in a beta state) for quite a while. --- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/

Nick Coghlan

11:59 p.m.

On 10 Jan 2014 02:05, "Giampaolo Rodola'" wrote:

...

On Thu, Jan 9, 2014 at 5:50 PM, Nick Coghlan wrote:

...
On 9 Jan 2014 22:03, "Giampaolo Rodola'" wrote:

...
On Thu, Jan 9, 2014 at 12:16 PM, Nick Coghlan

...

...
...
...
On 9 Jan 2014 09:49, "Amber Yust" wrote:

...
Also note that even if publicly visible projects are outnumbered by

...

...
...
...
It also mistakenly assumes our goal is to get existing *applications*

to migrate. It really isn't - we're obviously delighted if app developers choose to switch (as it indicates we have created a compelling platform), but we *needed* key library and framework developers to add Python 3 support in order to bootstrap the Python 3 development ecosystem.

True. I think one of the key points here is that different important libs haven't been ported yet: https://python3wos.appspot.com/ Too many of them are still marked red and IMO that is the main reason why a lot of people are being so hesitant, not unicode. "boto" alone counts as hundreds of thousands potential users which simply cannot migrate. Django made the transition only a couple of months ago, which basically means it's still in a beta state, and AFAIK fundamental projects such as Twisted don't even have an ETA. Considering 5 years have passed since Python 3.0 first made it's appearance I consider this a *serious* delay. From a user standpoint this sort of appears as a signal which

...

...
...
That's likely to apply even if project X is not within the list of your dependencies, because you may not depend from X now but maybe you will in the future, either because you need X or because Y requires X in order to work. It is *crucial* for people maintaining those libraries to put Python 3 porting on top of their TODO list at the cost of not working on new features.

This is still focusing on migrating *existing* applications.

I was talking about existing third party libraries (Twisted, gevent, lxml etc), not user applications. In order to port user applications you need those libraries to be ported first, and it is crucial that at least the most used ones are ported.

...
A user that starts with Python 3 simply wouldn't consider a dependency

wrote: private projects, the public projects tend to have a much larger impact on the overall ecosystem, because they're used by many entities (whereas private projects are typically only used by a single entity given their nature). translates into "if neither big project X has migrated after 5 years why should I?". like boto as an option

...

Why not? Note that I picked "boto" just because it's the first in that

list.

...

...
and would reach for asyncio rather than Twisted for their explicit

asynchronous programming needs.

I would't be so sure about that. We're talking about two very mature and established projects, with tons of third-party components (see https://github.com/facebook/tornado/wiki/Links), each solving a common set of problems in their own way, which will likely continue to be used independently from asyncio (which is still in a beta state) for quite a while.

Yes, Python 3 will be an even *better* ecosystem as more of the Python 2 ecosystem becomes available. That is not in dispute. The point is that *most new software* should be able to find appropriate packages in Python 3 at this point in time, and also has access to modules like "python-future", which make it relatively straightforward to downgrade to Python 2.7 if you start in Python 3 and then find a Python 2 only library that you absolutely positively have to depend on. This means that those unported libraries aren't a reason to *start* a greenfield project in Python 2. "Python 3 by default" *also* doesn't mean "never any reason to start a new Python 2 project instead". Cheers, Nick.

...

--- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/

Nick Coghlan

4:42 p.m.

On 9 Jan 2014 06:02, "Steven D'Aprano" wrote:

...

On Wed, Jan 08, 2014 at 05:22:02PM +0100, Alejandro López Correa wrote: [...]

...
Anyway, I understand it is not a clean way to proceed, but something

along

...

...
these lines might be the only way to speed up the adoption of python 3

One assumption in this discussion, and the various related discussions on Reddit and other places, is that adoption of Python 3 is too slow and needs to be sped up. I don't believe this is true. I believe adoption is just right and exactly what should be expected.

Alex Gaynor wrote a blog post a week or so ago claiming that, five years since Python 3 was first released, everyone should have migrated by now and that since only "five percent" (a figure which I believe he pulled out of thin air) have migrated, Python 3 has been a failure.

Alex's numbers were real - they're based on user agent header analysis for PyPI downloads. However, the other readily available metric is python.orginstaller downloads, and those favour Python 3 (and that's even before we publish 3.4, which has nice additions like pip, statistics and asyncio). Alex is a smart guy, but I don't know how he managed to get "After 5 years, Python 3 should be more widely used than Python 2" (clearly unrealistic) from "After 5 years, Python 3 should be mature enough to be the default choice for new projects". The latter is what we actually said, and, allowing for the 6 month delay to replace 3.0's unusably slow IO stack in 3.1, still looks plausible given the updates coming in Python 3.4. That article was actually the one that made me realise my Q&A needed a few more questions and answers :)

...

I challenge that belief. I've been hanging around here and on the Python-Dev list for a long time, and while I can't find any official pronouncement, the sense has always been that Python 3 adoption will take ten years, not five. (That's my recollection -- if any of the core developers wish to correct me, please do.)

5 years to be the default for new projects, we never set a goal for overtaking Python 2 overall. This Q and the one after it are most directly relevant: http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an...

...

Rates of adoption are much, much higher than gossip on the Internet suggests. About 70% of the top 200 projects on PyPI support Python 3, and downloads of Python 3 are very healthy, possibly even higher than downloads of Python 2. On the tutor list, I see a significant number of beginners using Python 3.

All our discussions with distros and redistributors are also about *how* to manage the transition, rather than *if*. Red Hat providing commercial support for Python 3.3 as of last September is a *big* deal, particularly given this week's announcement about CentOS being adopted as an officially Red Hat sponsored project and the popularity of CentOS as a platform in the scientific community.

...

It seems to me that given the circumstances, Python 3 adoption is right where we should expect it to be half-way through a decade-long process. There will be a period at the start when hardly anyone will migrate, then a period of accelerating migration, which will accelerate further when the mainstream Linux distros start shipping Python 3 as their system Python (ArchLinux is not mainstream, but Fedora is planning the change), followed by a sudden rush in another four or five years when people realise that Python 2.7 becoming unmaintained is no longer a distant prospect but is about to happen.

Yup. I actually started adding a timeline to my Q&A: http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an...

...

For many people, waiting until the last minute is the most sensible thing that they can do. This gives time for the early adoptors to discover and iron out all the wrinkles and difficulties. Rather than approaching this as "Python 3 has been a failure, what can we do to save it?" we should be approaching this as "Python 3 has been a success, what lessons can we take from the early adoptors to make it even easier for the next wave of adoptors?"

We've definitely made some mistakes in the area of communications, though. In particular, we probably should have had something like my Q&A available as an authoritative information source from the beginning, instead of only creating it around the release of Python 3.3.

...

"from __past__ import spam" does not make it easier to adopt. It just makes it easier to *put off adopting*.

...
and minimise the risk of defection to other languages.

People threaten that, but it is an irrational threat. (Mind you, people do silly, irrational things every day.) If you think its hard to migrate from Python 2 to 3, when you get to keep 90% of your code base and most of the backward-incompatible changes are a few libraries that have been renamed and a handful of syntax changes, how hard will it be to throw away 100% of your code and start again with a completely different language?

Exactly. By the time 2.7 goes into security fix only mode, we will have been maintaining Python 2 and Python 3 in parallel for more than *8 years*. This is a deliberate choice on our part to allow plenty of time for users to decide to migrate on their own, rather than attempting to force them to migrate with the stick of a lack of support. Even after that, commercial Python 2.x support will be available until at least 2023, and likely longer. Cheers, Nick.

...

-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Stephen J. Turnbull

8 Jan 8 Jan

10:51 p.m.

Nick Coghlan writes:

...

Anyway, the problems impacting wire protocol developers are known, but it's been damnably difficult to get anything other than "I like Python 2 better" out of them when it comes to discussing possible *solutions*

Good to know you feel that way too, I thought I just missed a lot of important discussions. :-(

...

The main challenge is thus getting people to stop asking the question "How do we bring back the Python 2 text model" (which is never going to happen - we changed the model for a reason), and instead ask "What changes can be made to Python 3, such as introducing additional purpose specific types, to make it a better language for wire protocol development?".

After spending enough time on Inada-san's use-case to find a real problem with treating wire protocols as text, I've come to the conclusion that those really are the same question, though. Add even a little bit of binary handling to a database connection, and even though almost everything is actually just ASCII, the few things that aren't blow everything up and you want everything to be bytes. At that point you end up really wanting bytes to have pretty much everything str does except maybe unidata lookups!

Georg Brandl

9 Jan 9 Jan

12:11 a.m.

Am 08.01.2014 16:38, schrieb Nick Coghlan:

...

On 9 January 2014 01:05, Nick Coghlan wrote:

...
On 9 January 2014 00:57, Alejandro López Correa wrote:

...
Hi,

I'm new here. I am sorry if this idea has already been discussed, but I have not found a way to search this list (I am not used to mailing lists at all).

I've seen recently some discussion in reddit about python 2 vs python 3, and the slow adoption of the latter. I am proposing here pragmatic way to speed up the process of porting old code and thus solving the split in the community, that I believe it is a serious threat. It is not clean, not at all, but it might work: just give python 2 whiners what they [we] want, and do it using "from __past__ import", in a similar way "from __future__ import" is used.

Hi,

You may want to read through http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_an... before lending too much weight to ill-informed Reddit commentary.

My apologies, that was rather rude of me when you're offering to help (I'm irritable at the moment since I've deemed it necessary to spend a bunch of time over the past week updating my Python 3 Q & A rather than enjoying my Christmas holidays, working on Python 3.4 or, this week, enjoying linux.conf.au 2014).

Please know that we all love you a bit more for that :) Georg

Chris Angelico

8 Jan 8 Jan

8:47 p.m.

On Thu, Jan 9, 2014 at 1:57 AM, Alejandro López Correa wrote:

...

I am thinking both in language syntax like division behaviour, unicode, str, etc, and major library changes.

The point of the __future__ directive is to enable per-module changes, which are applied at compile-time. The __future__ features spanning the 2.x / 3.x gap are: division (changes the meaning of an operator) absolute_import (changes the way modules are searched for) print_function (ditches some language magic in favour of a function) unicode_literals (changes the meaning of unadorned quoted strings) In theory, division and unicode_literals could probably be the targets of a from __past__ directive, but there's little point. Change the code now, use the directive, and then when you move to 3.x, the directive does nothing. (The other two would be more of a problem - I doubt the code to make print a statement exists in Py3, and the complete rewrite of the import machinery would make old-style importing dubious. In any case, you probably don't want old-style importing.) The unicode and str (or str and bytes) types have been the subject of some other discussions here on python-ideas, so I recommend reading up on those threads; I won't try to reopen the discussion here. There've been quite a few suggestions made, several of which could be quite viable without even requiring interpreter changes. Library changes are definitely not something you'd want a "from __past__ import" statement for. That would be exceedingly messy. However, there are a number of wrapper modules that let you bury the 2-vs-3 differences; instead of importing module X_name_1 or module X_name_2, you simply import X from wrapper, and it'll automatically give you the one you need. That at least covers the cases where the APIs are the same and it's just the names that differ. When anything more than that has changed, it wouldn't be possible to use a per-module flag (as "from __future__ import" is) to change that anyway. Once you feel the push to change interpreters and execute the code under Python 3, it's best to make your code run properly under Py3, rather than try to hold onto the past. Straddle the gap by continuing to run a Py2 interpreter and progressively changing your code to use __future__ print_function and division, and to get the text/bytes distinction clear, and then the jump to Py3 will be way easier. ChrisA

3754

Age (days ago)

3755

Last active (days ago)

List overview

Download

19 comments

9 participants

participants (9)

Alejandro López Correa
Amber Yust
Andrew Barnert
Chris Angelico
Georg Brandl
Giampaolo Rodola'
Nick Coghlan
Stephen J. Turnbull
Steven D'Aprano

from __past__ import division, str, etc

Alejandro López Correa

Alejandro López Correa

Alejandro López Correa

Alejandro López Correa

tags

participants (9)

from past import division, str, etc