Mailman 3 A python bridge between versions - Python-ideas

newer
PEP-257: drop recommendation for...

A python bridge between versions

ian o

28 Feb 2014 28 Feb '14

1:26 a.m.

The most quoted reason for staying with python 2 is that some required library is not available to support python 3. All it takes is one legacy library module in python 2 to keep an entire project in python 2. Many significant projects (e.g web2py) continue in python 2 for this reason. Even projects that produce code that can work under either python 2 or python 3, are themselves trapped into only using python 2 features. This is really holding back python! You cannot just import python2 code into the python 3 interpreter as it is not compatible. But could this be solved by a new approach which is to treat python 2 as another language and not directly import the code but handle python 2 code with an an extension module, which acts as bridge between versions. I mean, python three program can access C/C++ code (which is not python3), why can't the access python 2 code by treating it like C/C++? This would require an extension C module called from python 3. How to do this is well known. This extension C module would itself call the required module library(s) from python2. Calling python from C is also a well known technique. Ptyhon can call C. C can call python. So python3 call C which calls python2. So in python3 imput <python2-module> # this usually won't work because of language changes becomes import python2 <python2-module> = python2.import('<python2-module>') The resultant imported module would appear as a c++ extension to python3 and in effect be a wrapper for the python2 module using the python2 extension to execute the code This would mean programs using python2 imports have the overhead of both interpreters running in memory and calls across the python3/python2 boundary are 'wrapped' as c++ calls that then activate the required code within python 2. In reality quite a small overhead on modern computers. python2.sys.path (within the 'python2' module) or similar would be needed for the separate import space running inside the python2 boundary. Callbacks back across the python3/python2 boundary back into python3 are a further complication but several possible solutions exist to deliver this. Not many library modules require callback functions to the library module itself so this not the central issue. There is some work for the person coding the python 3 app in terms of a different syntax for legacy module imports and potentially setting python 2 environment values (like python2.sys.path), but not a lot and the python3 is the new program being worked on as opposed to the legacy library which can remain untouched. I think this type of approach could change the recommendation on 'python 2 or python 3' to a simple 'get the latest available to you and here is how to use legacy libraries if you have the need'. Thoughts or criticisms?

Attachments:

attachment.htm (text/html — 4.1 KB)

Show replies by thread

M.-A. Lemburg

28 Feb 28 Feb

1:51 a.m.

On 28.02.2014 00:26, ian o wrote:

...

[... Embed Python 2 in Python 3 ...]

Thoughts or criticisms?

There's a catch here: Python 2 and Python 3 use the same C APIs, so you'd have to separate the two in some way to make both live in the same process. It's not impossible, but it can potentially ruin the idea, since C extensions for both Python versions will have to link the right set of C APIs. What you can do right now is experiment with Pyro to use Python 2 and 3 in two different processes and have Pyro bridge between the two: http://pythonhosted.org/Pyro4/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

James Powell

2:48 a.m.

On 02/27/2014 06:51 PM, M.-A. Lemburg wrote:

...

On 28.02.2014 00:26, ian o wrote:

...
[... Embed Python 2 in Python 3 ...]

Thoughts or criticisms?

...

There's a catch here: Python 2 and Python 3 use the same C APIs, so you'd have to separate the two in some way to make both live in the same process.

I actually have a working embedding of Python 3 into Python 2, which I've presented at a few conferences, most recently at PyData London this past weekend. It's just a C-extension module that embeds a Python 3 PyRun_SimpleString. I haven't gotten around to building a shim to interact with Python 3 objects in Python 2 (and this would require a little bit of sophistication to handle GIL, GC, &c. issues.) Still, it's a working example of Python 2 and Python 3 running in the same process.

...

It's not impossible, but it can potentially ruin the idea, since C extensions for both Python versions will have to link the right set of C APIs.

I did this by "source filtering" - i.e., rewriting exported symbols to allow linking against both interpreters. This is admittedly a very primitive approach and limits its use in production. I recently put some time into trying to redo this embedding using cffi and worked through a few problems in this approach with one of the cython/numba developers. I haven't gotten anything working yet. I may need to ask Xzibit if he has any suggestions. I don't know if this idea has any serious applications. It's definitely been a fun toy for presenting ideas about interpreter embedding and exploring certain facets of CPython! Cheers, James Powell follow: @dontusethiscode + @nycpython attend: nycpython.org read: seriously.dontusethiscode.com

Ian

4:47 a.m.

On 28/02/2014 11:48 AM, James Powell wrote:

...

There's a catch here: Python 2 and Python 3 use the same C APIs, so you'd have to separate the two in some way to make both live in the same process. I actually have a working embedding of Python 3 into Python 2, which I've presented at a few conferences, most recently at PyData London this past weekend.

It's just a C-extension module that embeds a Python 3 PyRun_SimpleString. I haven't gotten around to building a shim to interact with Python 3 objects in Python 2 (and this would require a little bit of sophistication to handle GIL, GC, &c. issues.)

Still, it's a working example of Python 2 and Python 3 running in the same process. I don't know if this idea has any serious applications. It's definitely been a fun toy for presenting ideas about interpreter embedding and exploring certain facets of CPython!

Cheers, James Powell

James, I would suggest there is a very significant application for for calling python 2 objects from python3, in that suddenly the main roadblock to migration to python 3, the dependency on legacy python2 modules, could be removed as a roadblock. Certainly well worth working through the issues. Ian

Chris Angelico

6:40 a.m.

On Fri, Feb 28, 2014 at 10:26 AM, ian o wrote:

...

Ptyhon can call C. C can call python. So python3 call C which calls python2.

That's all very well if all you want to do is execute code. But as soon as you want to transfer data across, you have to do a whole lot of careful rummaging. First off, you'll need to have some C-level module that imports a Py2 module, identifies all its exported symbols (technically all its symbols, but you could save some effort by just looking at __all__ and most people won't see the difference), and creates a buffer layer between that and Py3: whenever someone calls up one of those symbols, create the translation and send it along. You can't have the two interpreters sharing GC state. But second, you have to deal with the fundamental data type differences. It may seem obvious enough - a function call goes through C and eventually becomes a function call, and so on - but the two interpreters fundamentally differ. Start with the easy one: int versus long. Py2 has two integer types; Py3 dropped int and renamed long to int. You need to traverse that gap. And then the big one. The really really big one. Bytes versus Unicode. How are you going to share strings across the boundary? Will all Py2's quoted strings come out as bytes in Py3? Will you encode and decode across the boundary (and if so, what encoding?)? It's fundamentally *hard* to do this. And there's another problem, too - a philosophical one. Let's suppose you put in all this work to make spamify.py available under Python 3. Your application, which needs spamify, can now migrate; but where's the impetus for spamify itself to start supporting Py3? Now that this shim is available, they can just carry on forever, right? So as soon as you create this, you effectively doom yourself to indefinite support, which is a *lot* of work. A new version of spamify might add a new feature, which you then need to support. Or it might start using a different encoding for its strings, and suddenly breaking everything. You have to keep up with all of that, *and* you're encouraging libraries to stay on Py2. In effect, you're encouraging applications to move to Py3, but encouraging libraries to stay on Py2. Personally, I'd rather people lean on their library creators to start supporting Py3. In the long run, it's going to be less work anyway. The only person who truly knows how a module should translate Py2 strings into Py3 strings is that module's author. ChrisA

ian o

8:03 a.m.

Chris, great reply. Yes it is not easy. Especially if you wish to make use of python2 modules painless. And I agree wholeheartedly that serious thought is needed on the 'could it work so well that it actually slow the migration of library modules?'. This is an important debate to have. I suggest if the answer to 'can a specification be created that will significantly accelerate migration?', then it is worth the effort to deliver this. But could it accelerate migration to python 3? Currently, the python.org website advice on : 'Python2 or Python3 - Which version should I use?' Lists two reasons for using python 2. This 'shim' could eliminate that second very significant reason. Then the advice would be 'if you have a choice of version, use python 3. And I suggest the topic of 'how do I use legacy python2 library modules?', would start by recommending if at all possible, find a python 3 replacement. As things stand, the popularity of Python has dropped by most measures since the release of Python 3. Improve the language and lose market share? I suggest that is all about the pain of the transition. You do not have to search hard to find comments to the effect 'nobody is using python3'. Or 'even if you use python 3, do not use the new features since you need to ensure code is compatible with both versions'. This is worth careful thought, and again I suggest part of the answer lies in the specification. If the 'do it all' approach to a 'shim' is taken, and using python2 modules is completely painless, no one will ever bother replacing them. Make it too hard to use python2 from python3 and migration is slowed again people do not bother to port libraries because not enough users have yet migrated. On python.org survey (2013-2014) , 60% of respondents report that dependencies are keeping them still working on python 2 and 80% are doing more code in python 2. For most people right now... all the changes and improvements made in python 3 are simply not available and the language is frozen. If something like this is done, without making it so painless that there is no incentive to replace or convert dependencies in a new world where the main reason for staying in python 2 has been basically eliminated, then this could be huge benefit. Just being able to change the python.org web site to say 'use python 3 unless you have no freedom of what version runs on your environment' would be huge! .

Chris Angelico

10:30 a.m.

On Fri, Feb 28, 2014 at 5:03 PM, ian o wrote:

...

Yes it is not easy. Especially if you wish to make use of python2 modules painless.

It's fundamentally not going to be painless. You can either try to move the pain into the thunking module, or keep it in the application, but using a Py2 module is not painless. And I'm not sure it's even possible to move all the pain into the thunker - most Py2 code simply doesn't concern itself with the difference between text and bytes, so there's no automated way to create that difference. (And if the module already does care, then chances are it's easy to port it and officially support Py3, which will be far FAR easier than using the shim - way higher performance too.)

...

And I agree wholeheartedly that serious thought is needed on the 'could it work so well that it actually slow the migration of library modules?'.

This is an important debate to have. I suggest if the answer to 'can a specification be created that will significantly accelerate migration?', then it is worth the effort to deliver this.

But could it accelerate migration to python 3?

At best, it'll accelerate migration of applications, while slowing migration of libraries.

...

Currently, the python.org website advice on : 'Python2 or Python3 - Which version should I use?' Lists two reasons for using python 2. This 'shim' could eliminate that second very significant reason. Then the advice would be 'if you have a choice of version, use python 3. And I suggest the topic of 'how do I use legacy python2 library modules?', would start by recommending if at all possible, find a python 3 replacement.

Yeah. And if you can't find a Py3 replacement, it might even be worth helping the library to support Py3. If running it through 2to3 does the job, maybe that's all you need to do!

...

As things stand, the popularity of Python has dropped by most measures since the release of Python 3. Improve the language and lose market share? I suggest that is all about the pain of the transition.

What measures?

...

You do not have to search hard to find comments to the effect 'nobody is using python3'. Or 'even if you use python 3, do not use the new features since you need to ensure code is compatible with both versions'.

This is worth careful thought, and again I suggest part of the answer lies in the specification. If the 'do it all' approach to a 'shim' is taken, and using python2 modules is completely painless, no one will ever bother replacing them. Make it too hard to use python2 from python3 and migration is slowed again people do not bother to port libraries because not enough users have yet migrated.

The problem is that there'll still be pain for the application, and there still won't be any for the library. The pain won't move at all. The impetus for a library to support Py3 comes from application developers saying "We need this to support Py3" (and, quite possibly, putting in the work to make it support Py3). Having a fiddly solution will encourage you to make the fiddly one work better, rather than seek the true one.

...

On python.org survey (2013-2014) , 60% of respondents report that dependencies are keeping them still working on python 2 and 80% are doing more code in python 2.

It's worth noting that that survey asked how many people were using Py2, not about how many weren't using Py3. Lots of people are using both. What that really means, then, is that 20% of those people are adding absolutely no new Py2 code. Quite a lot of those 80% are probably writing 2/3 compatible code.

...

For most people right now... all the changes and improvements made in python 3 are simply not available and the language is frozen.

Which means the "temptation gap" is getting progressively wider. The temptation to move from 2.4 to 2.6 is "hey look, you get the with statement", the temptation to move from 3.3 to 3.4 is "oh look, enumerations", and so on. The temptation to move to the current version is the sum of all the temptations from your current version onward. Sooner or later, that's going to be a sufficiently-strong incentive that people will just make the jump.

...

If something like this is done, without making it so painless that there is no incentive to replace or convert dependencies in a new world where the main reason for staying in python 2 has been basically eliminated, then this could be huge benefit.

Just being able to change the python.org web site to say 'use python 3 unless you have no freedom of what version runs on your environment' would be huge!

That's never really going to happen, partly because it's not very helpful :) Unless you're running on a locked-down server that has Py2 installed and no Py3, the problem isn't "no freedom what runs" but "the version suite I want doesn't exist". So the solution is to make what you want exist. In the open source world, there are two ways to do that: lean on the developers, or do the work yourself. (In the closed source world, you lose the second option, but on the flip side, it's that much more likely that your "lean" carries the weight of money.) A number of library/framework maintainers have said that the reason for not supporting Py3 is "lack of interest", which means they quite probably *would* support it if someone expresses enough interest to do the work of porting (or at least help with it); the more applications developers who say "I'd use Py3 except that libspamify.py needs Py2", the more likely that libspamify will sprout Py3 wings. Binaries of the latest Python 3 are available for most of the popular OSes. I don't keep track of Macs, but on Windows, it's easy to just grab a 3.3 (or, soon, a 3.4), and most Linux distros are carrying at least some form of 3.x (Debian's current stable, Wheezy, ships with 3.2, but Jessie ships with 3.3; RHEL 6 seems to have 3.3, but without having a copy, I can't check authoritatively). So it really is just a matter of library support. ChrisA

M.-A. Lemburg

11:16 a.m.

On 28.02.2014 01:48, James Powell wrote:

...

On 02/27/2014 06:51 PM, M.-A. Lemburg wrote:

...
On 28.02.2014 00:26, ian o wrote:

...
[... Embed Python 2 in Python 3 ...]

Thoughts or criticisms?

...
There's a catch here: Python 2 and Python 3 use the same C APIs, so you'd have to separate the two in some way to make both live in the same process.

I actually have a working embedding of Python 3 into Python 2, which I've presented at a few conferences, most recently at PyData London this past weekend.

It's just a C-extension module that embeds a Python 3 PyRun_SimpleString. I haven't gotten around to building a shim to interact with Python 3 objects in Python 2 (and this would require a little bit of sophistication to handle GIL, GC, &c. issues.)

Still, it's a working example of Python 2 and Python 3 running in the same process.

Interesting :-) Do you have some pointers to slides or videos ? I'm been thinking of doing something like this but the other way around - embed Python2 in Python3. When starting to brainstorm that idea, I quickly ended up postponing the idea again due to problems with C extension linking, dual interpreter environments, object interfacing between the two worlds, having two separate exception class trees, two sets of basic types/classes, two GILs, etc. The linking part was the most important to me, since being able to use Python 2 extensions would be my main reason to stick with Python 2 for some more time.

...

...
It's not impossible, but it can potentially ruin the idea, since C extensions for both Python versions will have to link the right set of C APIs.

I did this by "source filtering" - i.e., rewriting exported symbols to allow linking against both interpreters. This is admittedly a very primitive approach and limits its use in production.

I recently put some time into trying to redo this embedding using cffi and worked through a few problems in this approach with one of the cython/numba developers. I haven't gotten anything working yet. I may need to ask Xzibit if he has any suggestions.

I don't know if this idea has any serious applications. It's definitely been a fun toy for presenting ideas about interpreter embedding and exploring certain facets of CPython!

Cheers, James Powell

follow: @dontusethiscode + @nycpython attend: nycpython.org read: seriously.dontusethiscode.com

Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Chris Angelico

2:24 p.m.

On Fri, Feb 28, 2014 at 8:16 PM, M.-A. Lemburg wrote:

...

I'm been thinking of doing something like this but the other way around - embed Python2 in Python3. When starting to brainstorm that idea, I quickly ended up postponing the idea again due to problems with C extension linking, dual interpreter environments, object interfacing between the two worlds, having two separate exception class trees, two sets of basic types/classes, two GILs, etc.

The another common solution is to make the two halves completely language-independent. Separate two processes with a pipe, socket, etc, and communicate with streams of bytes moving back and forth. I'd consider that to be the zero-mark for any blending solution: it's simple, it's straight-forward, it's guaranteed to work, so you have to make something that's better than that - faster, easier to use, cleaner, whatever. That's the mark to beat. ChrisA

Nick Coghlan

3:18 p.m.

On 28 February 2014 18:30, Chris Angelico wrote:

...

On Fri, Feb 28, 2014 at 5:03 PM, ian o wrote:

...
Yes it is not easy. Especially if you wish to make use of python2 modules painless.

It's fundamentally not going to be painless. You can either try to move the pain into the thunking module, or keep it in the application, but using a Py2 module is not painless. And I'm not sure it's even possible to move all the pain into the thunker - most Py2 code simply doesn't concern itself with the difference between text and bytes, so there's no automated way to create that difference. (And if the module already does care, then chances are it's easy to port it and officially support Py3, which will be far FAR easier than using the shim - way higher performance too.)

...
And I agree wholeheartedly that serious thought is needed on the 'could it work so well that it actually slow the migration of library modules?'.

This is an important debate to have. I suggest if the answer to 'can a specification be created that will significantly accelerate migration?', then it is worth the effort to deliver this.

But could it accelerate migration to python 3?

At best, it'll accelerate migration of applications, while slowing migration of libraries.

And we don't actually mind all that much if applications don't migrate in the near term - Python 2 will have commercial support available well past 2020. (The developers of *those applications* might mind, though, just as anyone maintaining Python 2.4 compatibility for the benefit of RHEL/CentOS 5 typically isn't happy about it) By far the path of least resistance if developers would like to write Python 3 code, but are relying on some legacy Python 2 modules, is to instead write "Python 3 like" code in Python 2. Most of the interesting new Python 3 standard library modules (even asyncio) have Python 2 backports available on PyPI, and python-future (http://python-future.org/index.html) provides backports of the core data types that otherwise don't have Python 2 counterparts. This isn't *as* nice as actually running under Python 3 (you miss out on exception chaining for one thing, as well as the improvements to error messages in various parts of the standard library, and the infrastructure work that has gone into improving the core interpreter), but it's still a pretty nice environment to work in (heck, Python 2.*6* is still a pretty nice language to work in, although it definitely relies on more external library support than 3.x, or even 2.7). The group we *do* really care about supporting is authors of existing Python *libraries*, and "2in3" and "3in2" approaches don't really help them at all, since library and framework authors with users on both Python 2 and Python 3 will likely face demand to support both versions natively anyway. That's where various Python 3 changes like restoring Unicode string prefixes, restoring the binary transform codecs, and (for 3.5 in 2015) likely restoring binary interpolation support come in - by making the common subset of Python 2 and Python 3 larger, we make it easier for library authors to support Python 3 without breaking things for their existing Python 2 users. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

ian o

1 Mar 1 Mar

5:09 a.m.

...

And we don't actually mind all that much if applications don't migrate in the near term - Python 2 will have commercial support available well past 2020. (The developers of *those applications* might mind, though, just as anyone maintaining Python 2.4 compatibility for the benefit of RHEL/CentOS 5 typically isn't happy about it)

Your response is most dissapointing, and I would hope does not represent

the community overall. We have 3 teams working server systems currently with python as the implementation model. All work is currently in python 2.x and with a solution to this the work could immediately move to 3.x. However, you state that our situation, and that of the other 60% of python programmers as at end January 2014 who are constrained from using python 3 by python 2 dependencies: "we don't mind about those developers" And of course the dependent library module developers will not update the library, because 'no one out there uses python 3.x'. And from your perspective this is simply not of any interest. No wonder Tiobe index shows python dropping and in the words of those trying to persuade me to move the projects to another language 'python 3.x' came out 5 years ago and we still cannot move to it, this is going no where. I would like to still have the developers work in python, and to do that there needs to be a path to python 3. I also would prefer to be in python 3 anyway. But even if a workable solution is there, your answer is who cares about 60% of developers anyway!

Nick Coghlan

5:57 a.m.

On 1 March 2014 13:09, ian o wrote:

...

...
And we don't actually mind all that much if applications don't migrate in the near term - Python 2 will have commercial support available well past 2020. (The developers of *those applications* might mind, though, just as anyone maintaining Python 2.4 compatibility for the benefit of RHEL/CentOS 5 typically isn't happy about it)

Your response is most dissapointing, and I would hope does not represent the community overall.

My apologies, I wrote my post assuming you had already read http://docs.python.org/dev/howto/pyporting.html and decided that the existing approaches described there weren't sufficient for your purposes. Those cases where the recommended migration process isn't considered acceptable for some reason (usually non-technical ones) are the cases that I consider to fall into the "it's OK to just keep using Python 2" category. However, I just realised two things: * that version of the guide is currently only in the Python 3.4 docs, and so the page at http://docs.python.org/3/howto/pyporting.html is still displaying the older 3.3 version of the guide (the page in the Python 2.7 docs is similarly outdated). * that guide is currently aimed primarily at library and framework authors, and doesn't explicitly discuss the migration of integrated applications. I have now filed http://bugs.python.org/issue20812 and http://bugs.python.org/issue20813 to address those limitations, but in the meantime, I will provide the missing information here: For developers of integrated applications that currently still have some dependencies on Python 2, the preferred migration path is to use tools like python-modernize or python-future to shift first into the large common subset of Python 2 and Python 3, and then only later switch fully to Python 3. This approach permits application developers to take the following path: 1. Python 2 only (status quo) 2. Python 2/3 compatible on Python 2 (waiting for dependencies) 3. Python 2/3 compatible on Python 3 (dependencies ported or replaced) 4. Python 3 only (drop Python 2 support) Brett Cannon's "caniusepython3" tool (https://pypi.python.org/pypi/caniusepython3/) is designed to automate the dependency analysis to see if all declared dependencies are Python 3 compatible (or have suitable alternatives available). However, if you're using system packages for dependency management, some data transformations will be needed to convert them to a form that the tool understands. My original reply assumed that you were already aware of the details of this preferred approach before posting your proposal, and I now realise that assumption was likely incorrect. Once again, my apologies. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Steven D'Aprano

6:07 a.m.

On Thu, Feb 27, 2014 at 10:03:42PM -0800, ian o wrote:

...

Chris, great reply.

Yes it is not easy. Especially if you wish to make use of python2 modules painless.

And I agree wholeheartedly that serious thought is needed on the 'could it work so well that it actually slow the migration of library modules?'.

I think that such a bridge will slow the migration of library modules, since it reduces the need to migrate. As a library author, why should I spend tens or hundreds of hours migrating my library to Python 3 when I can spend 10 minutes adding a note to the library's website telling people to use the bridge? The problem with this suggestion is that it does nothing to encourage library authors to migrate to 3, it simply enables them to delay migrating even longer, possibly for so long that they lose interest altogether. That's library authors. How about the developers who use those libraries? Right now, there is some motivation for people to move to 3: cool new features, and getting cooler by the year. With this suggested bridge, they can migrate to 3 while still remaining dependent on 2. This might seem like a win if you only focus on the short term uptake of 3, but it misses the fact that they're still dependent on 2. They've now *trebled* the number of language dependencies, from Python only to two versions of Python plus the bridge software. Having spent the time and effort to migrate to 3, in a few years they'll have to go through the same pain again to get rid of the dependency on 2 and the bridge. Nobody will want to do that. The pressure on the core developers to keep supporting 2 forever will increase, not decrease.

...

This is an important debate to have. I suggest if the answer to 'can a specification be created that will significantly accelerate migration?', then it is worth the effort to deliver this.

I think you are making a classic mistake of technically-minded people: looking for a technical solution to a social problem. The problem with Python 3 uptake is not a technical problem, or at least not *only* a technical problem. It is mostly a social problem: the costs of migrating are greater than the benefits. Python 3 is great, but for most people Python 2 does the job. Another social problem is that the minority of library authors who haven't migrated haven't done so because they personally don't care about Python 3, or they don't have the time or money to spend on it. Adding a bridge doesn't suddenly give them more time or money. For application developers, the social problem is less because of a few libraries and more to do with the supported standard Python on their system. That will change once Linux distros start providing 3 as their standard system Python; out of the big distros, Fedora is about to do so, so things will start changing soon. But perhaps the biggest reason for slow uptake of Python 3 with existing projects is that nobody wants to spend money just to migrate to 3 for the sake of migrating to 3. "Written in Python 3" is not a selling point. Even the availability of security updates is not much of a selling point, although it will become more important when Python 2.7 is end-of-lifed. The reality is that many apps work perfectly well now using Python 2.7, or 2.6, or even 2.4, and they'll continue to work well in five or ten years so there's no real need to update. At the last US PyCon, there was even one fellow still using Python 1.5. It works, he didn't need security updates, so why migrate? *This is not a problem to be solved.* It's okay for people to stick with an old version of Python. There's no rule that says Python has failed if some developers stick to old versions.

...

As things stand, the popularity of Python has dropped by most measures since the release of Python 3. Improve the language and lose market share? I suggest that is all about the pain of the transition.

That sounds remarkably like FUD to me. Where is your evidence that the popularity of Python has fallen? Personally, I think all the "language popularity" measures are dubious, or at least need to be treated with considerable skepticism. Take this one: http://langpop.com/ which lists BrainF--- in the 40 most popular languages. Sure it is. The PyPL popularity index here: https://sites.google.com/site/pydatalog/pypl/PyPL-PopularitY-of-Programming-... shows Python as increasing in popularity. In contrast, TIOBE shows Python as losing popularity -- but so did Java, PHP, C++ and Ruby, and none of them have a major version change. (TIOBE also shows that despite a large number, possibly majority, of VB 6 developers boycotting VB.Net, VB.Net has just entered the top 10 most popular languages.) And then there's CodeEval, which for the third year in a row has found that Python is more popular than Java, C++, C, PHP and Javascript. In fact more popular than Java, C and PHP *together*. http://blog.codeeval.com/codeevalblog/2014 Bless their little white socks, but methinks their methodology is perhaps a tad flawed. I think it is interesting to compare and contrast these different ways of measuring popularity, but then I'm a statistics wonk and I find even flawed statistics interesting. Remember that there is no one true definition of popularity, even if there were there is no accurate way of measuring it, and all of the sites that claim to do so are actually measuring subtley different things. Most importantly, chasing popularity for its own sake is a waste of time. What makes Python Python is not just the syntax and std lib, but also the culture -- we're neither the stultifying corporate culture of "Java shops", nor the wild-west anything goes of cowboy PHP developers, but something unique. (As are other languages, of course. Perl culture is not Haskell culture is not Erlang culture.) In part, people take up languages because they like the culture. Python will never attract the corporate suits who like the Java philosophy, nor will it attract the cowbody web developers who like the PHP way of doing things. Should we try to ape PHP or Java to attract the cowboys or suits? Or should we continue to make Python the best language that *we* like, regardless of what other people prefer? Untimately, apart from winning pissing contests on Slashdot and Reddit, what does it matter if Python is third most popular or ninth most popular? Its not a popularity context where the winner takes all. According to TIOBE, the most popular language, C, has less than 20% share. If you think Python is a failure because it is in position 8, what does that make Javascript in position 9?

...

You do not have to search hard to find comments to the effect 'nobody is using python3'. Or 'even if you use python 3, do not use the new features since you need to ensure code is compatible with both versions'.

You don't have to search very hard to find comments that Elvis was kidnapped by aliens. https://duckduckgo.com/html/?q=elvis+kidnapped+by+aliens A belief which has rather more going for it than either of the two comments you list above, since there's a miniscule, microscopic chance that perhaps Elvis *was* kidnapped by aliens, whereas I know for a fact beyond any doubt that some people are using Python 3 and using the new features. -- Steven

ian o

6:10 a.m.

...

And we don't actually mind all that much if applications don't migrate in the near term - Python 2 will have commercial support available well past 2020. (The developers of *those applications* might mind, though,

.....

...

The group we *do* really care about supporting is authors of existing Python *libraries*, and "2in3" and "3in2" approaches don't really help them at all, since library and framework authors with users on both Python 2 and Python 3 will likely face demand to support both versions natively anyway.

Cheers,

...

Nick.

Nic,

Lets assume for the moment that the 'we don't really care about the 60% of python developers blocked from moving to python 3 by depencies' was not really what you meant. I am going to assume that what you really meant was 'I do not think helping these people directly is the best solution'. And this attitude is based around the fear that if the developers can move to python 3 before the libraries all support python 3, then that will remove pressure from the libraries to support python 3. This fear i can understand. But I suggest some real world experience shows this approach is actually not assisting either group. It is certainly not assisting the developers wishing to move to python 3, but it is also making life hard for the "people we do really care about" So not supporting a "2in3" solution forces programmers to with python2 dependencies to hold off moving to python3 until their dependencies migrate, but hopefully with sound motives. Now take a typical dependency. Web2py. Speak to them on migration and they will telly you 'none of our users have moved to python 3'. So the users are held back from moving by the dependencies, and the dependencies are held back by the users. I would suggest that making it easier to support both python2 and ptyhon3 from a library is simply reducing what it a painful thing. I would suggest that allowing all the end users to migrate to python3 as soon as possible is a far more attractive solution.

Andrew Barnert

7:09 a.m.

On Feb 28, 2014, at 19:09, ian o wrote:

...

However, you state that our situation, and that of the other 60% of python programmers as at end January 2014 who are constrained from using python 3 by python 2 dependencies: "we don't mind about those developers"

You're twisting the numbers. I'm part of the 60% who have to work on Python 2.x for at least one project. But first, I have others--including shipping commercial code both on the desktop and the server--using 3.x. And second, in all but one case, the 2.x apps are 2.x not because of lacking library support, but because we have an already-working and -deployed app that's effectively in maintenance mode, and it's not worth the effort or risk to port it to 3.x. (In other words, most of my 2.x work points to the _success_ of the migration plan, not a failure.) And you're also twisting Nick's words. He said that he cares more about getting library developers to 3.x than application developers. And he explained why that makes life better for application developers. To take that as "the Python devs don't care about application developers" is willfully misinterpreting the point.

...

And of course the dependent library module developers will not update the library, because 'no one out there uses python 3.x'.

None of the library devs I've ever dealt with has ever said that (well, not since the early 3.1 days). Most projects that don't support 3.x today want to do so, it just isn't always the very highest thing on their priority list. Three times I've had to port a library myself rather than waiting around--but in every case, the library's dev was happy to take my code, even if it meant losing 2.5 support, reviewing large changelists, etc. I'm sure there are some exceptions, but it's certainly not the ubiquitous case you're presenting. There's a reason the Python 3 Wall of Shame changed it's name to Wall of Superpowers and the competing py3ksupport page stopped updating: because people have been porting important libraries to 3.x, and continue to do so.

...

No wonder Tiobe index shows python dropping

And is the Python 3 migration also to blame for Python's direct competitors (Ruby, Perl, and PHP) also dropping)? Or for the general trend in Tiobe numbers to show ObjC and Windows-specific languages continually gaining on cross-platform languages?

...

and in the words of those trying to persuade me to move the projects to another language 'python 3.x' came out 5 years ago and we still cannot move to it, this is going no where.

This is pure FUD. And I don't see how repeating that FUD to the Python devs is going to help you. If you want to convince these people to stay on Python, you need to start talking specifics instead of talking points. What libraries are stopping you from moving to 3.x? Does Ruby or Node.js or whatever actually have a better alternative for all of those libraries than Python 3 does? Would it really be easier to write those missing libraries yourself in a new language than to port existing Python 2.x libraries to 3.x? For that matter, have you actually checked whether those libraries really are blocking you? I've lost count of the number of people who've claimed they can't switch to Python 3 because of some library that's been ported since 2010 or that has a 3.x-compatible drop-in replacement.

ian o

7:53 a.m.

Steven, Great post. Arguments can be enjoyed when people with differing views argue them well. Let me try and put the counter case as well. I have to start with the Elvis abducted by aliens just because it is such a great place to start. I enjoyed your point. But let me counter by saying that the information that some people say 'Elvis was abducted by aliens' does not mean this is what actually happened, but it does tell us that some people believe this! And there is rather surprising information in the fact that 'some people believe Elvis was abducted by Aliens'. Similarly I do not believe 'no-one is using python 3'. I program in python3 (and wish i could for all programming). But it is still interesting that people are saying it. Other comments inline below: I think that such a bridge will slow the migration of library modules,

...

since it reduces the need to migrate. As a library author, why should I spend tens or hundreds of hours migrating my library to Python 3 when I can spend 10 minutes adding a note to the library's website telling people to use the bridge?

The problem with this suggestion is that it does nothing to encourage library authors to migrate to 3, it simply enables them to delay migrating even longer, possibly for so long that they lose interest altogether.

I suggest the counter point here is to consider life a library developer. Which motivates you: that fact that people using your library all program in python3, or hearing that all your users program in python2?

...

That's library authors. How about the developers who use those libraries?

Right now, there is some motivation for people to move to 3: cool new features, and getting cooler by the year. With this suggested bridge, they can migrate to 3 while still remaining dependent on 2. This might seem like a win if you only focus on the short term uptake of 3, but it misses the fact that they're still dependent on 2. They've now *trebled* the number of language dependencies, from Python only to two versions of Python plus the bridge software.

Are you assuming that it will always be either a) all dependencies require python2 or b) all dependencies are available in python 3. Rather than c) some dependencies are available in python3 but the python3 version cannot be used because I am being held back to python2 by some other critical dependencies. For the projects i work on, it is c). It is not like there has been no success getting packages to migrate. The problem is that an entire project is held back by one single dependency.

...

have to go through the same pain again to get rid of the dependency on 2 and the bridge. Nobody will want to do that. The pressure on the core developers to keep supporting 2 forever will increase, not decrease.

I cannot see that getting rid of the bridge would be that difficult,

And for us we would wear some pain right now and make that move. Most of the project doesn't have tow steps and can be moved immediately. If a bridge is offered, it would not be a great idea to force everyone to use it. If EVERY imported module is still in python2 then I suggest it is too early to move. Demonstrably, some people like the idea, and I suggest it would be interesting to discover how many. There may be cases where it is not the right choice because too many dependencies are only available in python3 , but alone that does not mean the choice should not be offered. Personally I beleive that now many dependencies can be found in python3, hence the frustration of having to wait until every single one moves seems unfortunate. Having spent the time and effort to migrate to 3, in a few years they'll particularly if it is not for many modules. The real pain is living in world where there is so much holding everything back to the older version. Again, let me declare the self interest here. We have development teams and projects we wish to move to python3. A bridge like this would allow us to move now, and that would allow pressure on the module providers. Stopping a step like this on the basis it stops people like us moving to python3 just seems counter productive if the goal is to move to python3.

...

...
This is an important debate to have. I suggest if the answer to 'can a specification be created that will significantly accelerate migration?', then it is worth the effort to deliver this.

I think you are making a classic mistake of technically-minded people: looking for a technical solution to a social problem.

The problem with Python 3 uptake is not a technical problem, or at least not *only* a technical problem. It is mostly a social problem: the costs of migrating are greater than the benefits. Python 3 is great, but for most people Python 2 does the job.

I am a person experiencing the problem. And our experience is that our

core developers use code supplied by a third party who does not produce a python3 version. So the core has to be done in python2. The code produced by these core developers is then supplied to other companies - forcing them to stay in python2. It is dependency hell. One module anywhere down the chain that cannot be moved and the whole project is forced to stay python2.

...

Another social problem is that the minority of library authors who haven't migrated haven't done so because they personally don't care about Python 3, or they don't have the time or money to spend on it. Adding a bridge doesn't suddenly give them more time or money.

Agreed. But in the real world, an example of a major dependency requiring python2, web2py, states they will be motivated to python3 when the users of their framework move. But this policy of forcing all dependencies to move before the end users move means this will never happen. What it does mean is many of their customers can move. And it is even possible for them to ask their holdout customers to either stay with the old version or move, since these customers are no longer held back by other dependencies. For application developers, the social problem is less because of a few

...

libraries and more to do with the supported standard Python on their system. That will change once Linux distros start providing 3 as their standard system Python; out of the big distros, Fedora is about to do so, so things will start changing soon.

But perhaps the biggest reason for slow uptake of Python 3 with existing projects is that nobody wants to spend money just to migrate to 3 for the sake of migrating to 3. "Written in Python 3" is not a selling point. Even the availability of security updates is not much of a selling point, although it will become more important when Python 2.7 is end-of-lifed. The reality is that many apps work perfectly well now using Python 2.7, or 2.6, or even 2.4, and they'll continue to work well in five or ten years so there's no real need to update. At the last US PyCon, there was even one fellow still using Python 1.5. It works, he didn't need security updates, so why migrate?

Perhaps the biggest reason ..... Why not ask? Well, python.org conducted a survey as of January 2014 an answer given was that 60% of developers were held back from moving projects because of dependencies. This is a big number and it is not just guessing but real figures.

*This is not a problem to be solved.* It's okay for people to stick with

...

an old version of Python. There's no rule that says Python has failed if some developers stick to old versions.

OK. So you are telling us forget moving our developers to python3 until every last dependency has moved. And we should also force the people who in turn use our code to stay with python2 and it is not a problem.

...

...
As things stand, the popularity of Python has dropped by most measures since the release of Python 3. Improve the language and lose market share? I suggest that is all about the pain of the transition.

That sounds remarkably like FUD to me. Where is your evidence that the popularity of Python has fallen?

Personally, I think all the "language popularity" measures are dubious, or at least need to be treated with considerable skepticism. Take this one:

http://langpop.com/

which lists BrainF--- in the 40 most popular languages. Sure it is.

The PyPL popularity index here:

https://sites.google.com/site/pydatalog/pypl/PyPL-PopularitY-of-Programming-...https://www.google.com/url?q=https%3A%2F%2Fsites.google.com%2Fsite%2Fpydatalog%2Fpypl%2FPyPL-PopularitY-of-Programming-Language&sa=D&sntz=1&usg=AFQjCNEJ__tv-fiB_BTXtt3qu6Y_UD49kg

Thanks for that- useful link! Please remember I am the converted, so it

...

shows Python as increasing in popularity. In contrast, TIOBE shows Python as losing popularity -- but so did Java, PHP, C++ and Ruby, and none of them have a major version change. (TIOBE also shows that despite a large number, possibly majority, of VB 6 developers boycotting VB.Net, VB.Net has just entered the top 10 most popular languages.)

And then there's CodeEval, which for the third year in a row has found that Python is more popular than Java, C++, C, PHP and Javascript. In fact more popular than Java, C and PHP *together*.

http://blog.codeeval.com/codeevalblog/2014 http://www.google.com/url?q=http%3A%2F%2Fblog.codeeval.com%2Fcodeevalblog%2F2014&sa=D&sntz=1&usg=AFQjCNEcIR2Zkg32DfKkhns8eq7KVgFJ1Q

Perhaps overstated, but it helps to have sources like this. It does say coding tests and challenges. I live in a country where school children have one national coding contest/challenge, and it is in python. (And in

helps to have arguments to push back with when others push tiobe in front of me. python 3, no dependencies :) ) Bless their little white socks, but methinks their methodology is

...

perhaps a tad flawed.

I think it is interesting to compare and contrast these different ways of measuring popularity, but then I'm a statistics wonk and I find even flawed statistics interesting. Remember that there is no one true definition of popularity, even if there were there is no accurate way of measuring it, and all of the sites that claim to do so are actually measuring subtley different things.

Most importantly, chasing popularity for its own sake is a waste of time. What makes Python Python is not just the syntax and std lib, but also the culture -- we're neither the stultifying corporate culture of "Java shops", nor the wild-west anything goes of cowboy PHP developers, but something unique. (As are other languages, of course. Perl culture is not Haskell culture is not Erlang culture.) In part, people take up languages because they like the culture. Python will never attract the corporate suits who like the Java philosophy, nor will it attract the cowbody web developers who like the PHP way of doing things. Should we try to ape PHP or Java to attract the cowboys or suits? Or should we continue to make Python the best language that *we* like, regardless of what other people prefer?

All agreed on the culture. But i suggest this is the biggest point. Stopping measures like a bridge that would allow many more end users to code in python3 just seems a negative culture. Effectively you are telling us that our team should not move to python3.

...

Untimately, apart from winning pissing contests on Slashdot and Reddit, what does it matter if Python is third most popular or ninth most popular? Its not a popularity context where the winner takes all. According to TIOBE, the most popular language, C, has less than 20% share. If you think Python is a failure because it is in position 8, what does that make Javascript in position 9?

...
You do not have to search hard to find comments to the effect 'nobody is using python3'. Or 'even if you use python 3, do not use the new features since you need to ensure code is compatible with both versions'.

You don't have to search very hard to find comments that Elvis was kidnapped by aliens.

https://duckduckgo.com/html/?q=elvis+kidnapped+by+aliens

A belief which has rather more going for it than either of the two comments you list above, since there's a miniscule, microscopic chance that perhaps Elvis *was* kidnapped by aliens, whereas I know for a fact beyond any doubt that some people are using Python 3 and using the new features.

I think I already covered this one. Again the point is that some people

say it. We both know it is not literally true. If i did not program in python3 I would not bother posting suggestions to try and allow more programming in python3. Ian

...

-- Steven _______________________________________________ Python-ideas mailing list Python...@python.org javascript: https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Steven,

Chris Angelico

8:43 a.m.

On Sat, Mar 1, 2014 at 4:53 PM, ian o wrote:

...

A bridge like this would allow us to move now, and that would allow pressure on the module providers.

Think about what the bridge really means. It means that you move now, yes, but it also means that the existing module is working fine. That means there is *no* pressure on the module provider. The pain of maintaining the bridge is all yours. You want to put pressure on the module provider? *Don't* have the bridge. Then you have an incentive to get that module ported, because then you could migrate. Maybe that means you lean on the developers; maybe that means you run the code through 2to3 and then start dealing with test suite failures, and then submit the code back to them and say "Here's Python 3.3 support, these are the downsides, will you accept it?". That's the sort of pressure that's likely to work. Saying "Hey, we can use your module in Python 3 now, thanks to a Python-in-C-in-Python bridge" will evoke the response "Okay fine, no problem then!". ChrisA

Nick Coghlan

8:55 a.m.

On 1 March 2014 15:53, ian o wrote:

...

Steven,

...
*This is not a problem to be solved.* It's okay for people to stick with an old version of Python. There's no rule that says Python has failed if some developers stick to old versions.

OK. So you are telling us forget moving our developers to python3 until every last dependency has moved. And we should also force the people who in turn use our code to stay with python2 and it is not a problem.

Ian, please keep in mind that you are asking people to do extra work for you *for free*, after you have already been benefiting for years from work that we already made available to you free of charge. The latter part is OK, and entirely the way non-commercial open source works, but it *does* mean that you *don't* get to tell the core development team our priorities are wrong when you haven't paid us a cent. If that's the kind of relationship you want, pay a commercial vendor to listen to your concerns - mediating between customers and the community one of the services commercial open source vendors provide (and the best ones will then contribute upstream development effort on your behalf). By contrast,community oriented open source operates on a time-based economy - the only ways for people to get things are done to dedicate their own time, inspire others to dedicate their time, or else to pay developers (either directly or indirectly through a vendor) to spend *their* time on projects of interest to those providing the funds. The CPython core development team has made it clear we don't want to support running Python 2 and 3 code in the same process because doing so is *hard* (and perhaps even impossible without completely redesigning the interpreter) and certainly *not fun*. The inspiration based approach is unlikely to work in this case (because we've already considered the possibility and dismissed it as impractical), but that still leaves open the possibility of finding someone else that is interested in attempting to prove us wrong, as well as people finding a way to pay someone to make it happen. However, in both cases, keep in mind that the people *most* familiar with the CPython code base think it's a bad idea that probably won't work. Instead, with the aid of many other members of the community, we've created a source based migration approach that involves taking advantage of the large common subset of Python 2 & 3. If commercial operations higher in the stack would like to migrate to Python 3, but are relying on dependencies that the community has provided for free, then it's time to think seriously about *investing* in the community and helping those projects to migrate (either by contributing developer time directly or by contributing funds to the existing developers for those projects). Note that the Python Software Foundation is able to help mediate requests for assistance with getting dependencies ported. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Andrew Barnert

9:44 a.m.

On Feb 28, 2014, at 21:53, ian o wrote:

...

Agreed. But in the real world, an example of a major dependency requiring python2, web2py, states they will be motivated to python3 when the users of their framework move.

This is an interesting choice for an example. web2py isn't just a library, it's a framework. Its main purpose is to run your code in the right situations. Which is exactly the kind of thing your bridge cannot handle, for the reasons you already gave. And I don't think this is a coincidence. The same thing is true of many of the most prominent packages that haven't been ported yet--Twisted, Paste, gevent, supervisor, etc. On top of that, web2py is all about passing Internet strings around. The same thing that makes it difficult to port--the bytes/Unicode distinction--would make it just as hard to use through the bridge. Anyway, if you're interesting in pursuing this, obviously nobody is going to stop you from doing it and putting it on PyPI. And if there's as much interest in the idea as you think, you should start getting outside contributions as soon as you have a workable beta (or you might even be able to get funded development before that). And if it became popular enough, you could come back to the core dev team and say, "Like it or not, there are tons of major projects relying on my bridge, and the consequences aren't nearly as gloomy as you suspected, so what about putting it in the stdlib?"

ian o

10:03 a.m.

...

Ian, please keep in mind that you are asking people to do extra work for you *for free*, after you have already been benefiting for years from work that we already made available to you free of charge. The latter part is OK, and entirely the way non-commercial open source works, but it *does* mean that you *don't* get to tell the core development team our priorities are wrong when you haven't paid us a cent.

Nic, Why are you so aggressive? No one is trying to 'tell' people to do something. I was proposing an idea here. I thought it was a relevant useful idea, so i have been arguing the case that idea is useful. Others argue the case that it is not useful and to me this is a constructive way to share ideas, hear other points of view so they can be considered. For commercial purposes you have managed to convince me I should cease to lobby for python to be used in the projects where I assist. Obviously commercial use is clearly not desired by the community, that is the first lesson i have learned. As I am not paid to champion python in the work environment, having been enlightened that python community does not desire commercial usage, I will cease. I came this forum hoping to contribute. I do not get paid to post here, nor will python being used in projects where i am involved generate a financial outcome for me. I simply feel a passion for the language and have been supporting the use of python in education and it helps push case for students learning python when there is also commercial usage. Is educational usage also to be discouraged? So now I am being told that not only are commercial users not to be encouraged, but also opinions and ideas of those who have been involved in commercial projects are not welcome. Disappointing. If a welcome as a result of a debate it was agreed that the approach I feel will help both library module developers and end users and was worth supporting but the community then the decision question would come as to who would donate time to implementation. While i would get no commercial benefit and do not have a lot time i would have tried to help. I think the idea has merit. But what is the point? It seems even debate on desirability of the idea in your opinion is not to be encouraged. From others I get the sense of community. But from you I get the sense of a closed club. Ian

Chris Angelico

10:05 a.m.

On Sat, Mar 1, 2014 at 7:03 PM, ian o wrote:

...

Why are you so aggressive? No one is trying to 'tell' people to do something.

I was proposing an idea here.

If your idea is accepted, someone has to implement it :) That's work, and a lot of it. So unless you're volunteering to actually write this thunker/bridge/shim/etc, you're asking someone else to. ChrisA

ian o

10:40 a.m.

Chris, First the idea has to be accepted as desirable, then feasible with an agreed specification. All to no avail unless someone can then implement it. I would volunteer to assist but i assume there have to be enough others to volunteer as well if it was any help. But there is no intent to tell people to do something. There has to be sufficient enthusiasm for anyone involved to actually want to do it. But I have been told that those who would benefit are people that there is no desire to help. My suggestion that it would also be helpful to people there is a desire to help seems to have been met with the response 'we do not want to hear this'. I guess I expected a more open community. I have learned a lot, but not what I expected. I am still not sure what the criteria is to actually be welcome here. I have very much appreciated your comments though. Ian On Saturday, March 1, 2014 7:05:52 PM UTC+11, Chris Angelico wrote:

...

On Sat, Mar 1, 2014 at 7:03 PM, ian o javascript:> wrote:

...
Why are you so aggressive? No one is trying to 'tell' people to do something.

I was proposing an idea here.

If your idea is accepted, someone has to implement it :)

That's work, and a lot of it. So unless you're volunteering to actually write this thunker/bridge/shim/etc, you're asking someone else to.

ChrisA _______________________________________________ Python-ideas mailing list Python...@python.org javascript: https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Georg Brandl

11:11 a.m.

Am 01.03.2014 09:40, schrieb ian o:

...

Chris,

First the idea has to be accepted as desirable, then feasible with an agreed specification. All to no avail unless someone can then implement it.

I would volunteer to assist but i assume there have to be enough others to volunteer as well if it was any help.

But there is no intent to tell people to do something. There has to be sufficient enthusiasm for anyone involved to actually want to do it.

But I have been told that those who would benefit are people that there is no desire to help. My suggestion that it would also be helpful to people there is a desire to help seems to have been met with the response 'we do not want to hear this'.

I guess I expected a more open community. I have learned a lot, but not what I expected. I am still not sure what the criteria is to actually be welcome here.

Hi Ian, I'm sorry you feel like this. It's definitely not the intention of this list to exclude people -- just look at the wild ideas being discussed all the time... The Python 2/3 topic is simply a "sore point" for many of us, having discussed it time and again, often times with quite aggressive people (with the "why have you ruined my Python" mindset). Therefore we tend to become a little categorical faster than with other topics, even to people like you who aren't being aggressive. I hope you can forgive Nick, who has put tons of work into the excellent 2 vs 3 FAQ, being a bit thin-skinned :) In other words: Welcome to the Python community - don't let the 16-ton weight scare you off! cheers, Georg

Nick Coghlan

11:46 a.m.

On 1 March 2014 19:11, Georg Brandl wrote:

...

Am 01.03.2014 09:40, schrieb ian o:

...
I guess I expected a more open community. I have learned a lot, but not what I expected. I am still not sure what the criteria is to actually be welcome here.

Hi Ian,

I'm sorry you feel like this. It's definitely not the intention of this list to exclude people -- just look at the wild ideas being discussed all the time...

The Python 2/3 topic is simply a "sore point" for many of us, having discussed it time and again, often times with quite aggressive people (with the "why have you ruined my Python" mindset). Therefore we tend to become a little categorical faster than with other topics, even to people like you who aren't being aggressive. I hope you can forgive Nick, who has put tons of work into the excellent 2 vs 3 FAQ, being a bit thin-skinned :)

Aye, I hit the "This is holding back Python!" in Ian's original post, and subsequently interpreted the rest of the post (and thread) in an extremely negative light. We get a *lot* of people making such accusations (understandably so, even though those accusations aren't adequately backed up by the available data), so I have a very low tolerance at the moment for anything that even hints at commercial operations complaining about the quality of open source work they aren't funding (or otherwise contributing to), but are relying on anyway. However, Ian, It sounds like that wasn't your intent, so my apologies (once again) for taking it that way. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Stefan Behnel

12:51 p.m.

ian o, 01.03.2014 04:09:

...

We have 3 teams working server systems currently with python as the implementation model. All work is currently in python 2.x and with a solution to this the work could immediately move to 3.x.

You are mixing things up here. If you are developing application code, you don't really have a problem. Either you stay with 2.x or you switch to 3.x. You may even decide to stay in Py2 with some of your applications and migrate the others. Or migrate one after the other, over months or years. You really don't have to break your running systems, and definitely not all of them at once. It's entirely your choice, and doing the switch (if you want to) really isn't all that hard. Nick pointed you to a couple of resources that can help you in the progress. Many, many others have done it before, and have found that breaking all bridges after you have passed over them is a perfectly reasonable and efficient way to migrate, *specifically* for application code. The real problem, and that's what Nick was referring to, is writing and maintaining libraries that others depend on, because these others may use Python 2.x (and may not even want to switch to Py3), or they may use Python 3.x (and do not care about Python 2.x). But the library authors do not control the application code. Therefore, to cater for both user groups, library maintainers currently have to write code that works on both platforms, which is annoying and time consuming. Since you seem to be interested in migrating your application code in order to take advantage of Python 3, my advice is to (ta-da!) start migrating your code. And when that's done, stop looking back. Any "solution" that involves running a mix of Py2 and Py3 inside of one application is just screaming for endless trouble. Don't forget that "temporary solutions" tend to be the most durable in practice. Stefan

Giampaolo Rodola'

1:33 p.m.

On Sat, Mar 1, 2014 at 6:09 AM, Andrew Barnert wrote:

...

For that matter, have you actually checked whether those libraries really are blocking you? I've lost count of the number of people who've claimed they can't switch to Python 3 because of some library that's been ported since 2010 or that has a 3.x-compatible drop-in replacement.

The problem you mention here is absolutely *real*. It's not only a matter of whether all your deps have currently been ported already. Software evolves over time and you can also remain stuck later, when you suddenly decide you need lib X and lib X hasn't been ported yet (uh-oh! what do I do now?). And please, note that *as of now* lib X includes names such as Twisted and Paramiko, which means hundreds of thousands of users which are stuck as we speak. How can a company decide to make the switch starting from such an assumption? The assumption that lib X or Y might not be ported soon, or even *ever*? IMHO as long as we won't get close to a 100% of libs being ported the current situation is not likely to change. FWIW here's some stats (inspired by recent Brett Cannon's post about openstack): http://stackoverflow.com/a/22113627/376587 -- Giampaolo - http://grodola.blogspot.com

Mark Lawrence

5:48 p.m.

On 27/02/2014 23:26, ian o wrote: I have read every entry so far on this thread and have been forced to the conclusion that this is A Bridge Too Far. Sorry :( -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com

Oscar Benjamin

7:34 p.m.

On 1 March 2014 03:09, ian o wrote:

...

...
And we don't actually mind all that much if applications don't migrate in the near term - Python 2 will have commercial support available well past 2020. (The developers of *those applications* might mind, though, just as anyone maintaining Python 2.4 compatibility for the benefit of RHEL/CentOS 5 typically isn't happy about it)

Your response is most dissapointing, and I would hope does not represent the community overall.

We have 3 teams working server systems currently with python as the implementation model. All work is currently in python 2.x and with a solution to this the work could immediately move to 3.x.

However, you state that our situation, and that of the other 60% of python programmers as at end January 2014 who are constrained from using python 3 by python 2 dependencies: "we don't mind about those developers"

Hi Ian, In the past I've sometimes seen Nick being slightly careless with his words with the result that someone misunderstands him and gets upset. That's not what's happening here though. I'm not sure if you're wilfully misunderstanding him or just unable to see this from his perspective. Nick clearly did not say that he (or anyone else) doesn't care about any particular group of *developers*. What he meant is that if you as an application author have decided that sticking with Python 2 is the right thing to do right now then he won't try to argue with you. If your application is currently working and has a dependency on web2py and it's not worth your time/money to port web2py yourselves then sticking with Python 2 may be the right decision for you. OTOH if you were the author of web2py and said "I don't want to port web2py to Python 3" Nick would mind. He would want to know why you didn't want to port to Python 3 and whether or not there was something (reasonable) that the core Python devs could do to help. This is because if a library is not ported then that has a knock-on effect on application authors (such as you) who would like to port an existing application or write a new application for Python 3. Oscar

Stephen J. Turnbull

2 Mar 2 Mar

4:37 p.m.

...

...
...
...
...
Ian O. writes: Nick Coghlan writes:

...

...
Ian, please keep in mind that you are asking people to do extra work for you *for free*, after you have already been benefiting for years

...

Why are you so aggressive? No one is trying to 'tell' people to do something.

In Nick's defense, (1) he spends a *lot* of his time explaining why Python development proceeds as it does, (2) "just proposing ideas" sometimes is indeed a passive-aggressive ploy, (3) your timing is bad due to the appearance of an occasional poster who does do exactly that, and (4) Nick's probably harried with a major release in process and PyCon coming up.

...

For commercial purposes you have managed to convince me I should cease to lobby for python to be used in the projects where I assist.

I think that's *exactly* the wrong lesson to learn, unless your management doesn't understand that free software is an investment and requires its users to spend resources to take advantage of it. It's just that the form of the resources required is often quite different from those required to use proprietary software. And of course it differs from product to product even within open source. So all you've learned from Nick is that Python is unlikely to dramatically reduce the resources required to migrate from Python 2 to Python 3 using technology like a "bridge". "Everyone would like a pony, but we don't have everyone's favorite pony." OTOH, it remains the case that most new applications can be written in Python 3 because there are good libraries to support them, although you may have to give up on some frameworks that depend on libraries that haven't been ported. That may be, but is not necessarily, a show-stopper. Now you say "lobbying". "Lobbying" implies effort and marshalling resources. Even if you strongly favor a particular framework that is *apparently* blocked by an unported library, it's surely theoretically possible that the shortest path to serving your company's software needs is spend company resources to port that library[1] -- as well as the personal resources to get the necessary support and resources from management! In practice, while I can't offhand give a case where a commercial company has done that port internally, I know at least one noncommercial project that is considering paying for a port of one of its Python3 blockers. I'm willing to bet you can find companies that have done 2to3 ports of libraries considering that to be the "shortest path to a product release", if you look around a bit.

...

Obviously commercial use is clearly not desired by the community, that is the first lesson i have learned. As I am not paid to champion python in the work environment, having been enlightened that python community does not desire commercial usage, I will cease.

Are you aware how passive-aggressive your phrasing sounds? I'm sure you're well aware that not only are many Python developers involved in commercial development, but all of Python from the license on up is oriented toward encouragement of commercial use of Python. And it's been quite successful in commercial use. It would be nice if the language improvements in Python 3 came at zero cost. They don't, and we all feel that pinch at one time or another. But remember, innovations succeed not only by inventing completely new ideas that nobody ever thought of before, but also by hurdling cost barriers that others considered so high as to make known "interesting in theory" ideas "commercially impractical". I am not going to tell you that your cost barrier is lower than you think -- you know your situation, I don't. But I assure you that others, with high but not insuperable cost barriers, have indeed made that hurdle in porting libraries to Python 3, and I'm sure some are taking on new challenges today. Footnotes: [1] Just as if it were a completely different language ("that Perl module would be perfect if only it were in Python..." :-).

Stephen J. Turnbull

4:55 p.m.

Stefan Behnel writes:

...

Since you seem to be interested in migrating your application code in order to take advantage of Python 3, my advice is to (ta-da!) start migrating your code. And when that's done, stop looking back.

I'm sure he's thinking that way for himself. To be fair, though, as he describes it, Ian is evidently not the boss in the commercial projects he's talking about projects. He needs to convince others that Python 3 is a practical solution to their problems. This makes the idea of lobbying for Python a real enthusiasm drainer. I'm still all for it, but we should recognize the social aspects as well as the technical issues. Unfortunately, I'm not sure what, if anything, we can do to help support developers like Ian who can't impose Python 3 -- but rather has to expend personal energy and reputation in lobbying.

3706

Age (days ago)

3709

Last active (days ago)

List overview

Download

29 comments

14 participants

участники (14)

Andrew Barnert
Chris Angelico
Georg Brandl
Giampaolo Rodola'
Ian
ian o
James Powell
M.-A. Lemburg
Mark Lawrence
Nick Coghlan
Oscar Benjamin
Stefan Behnel
Stephen J. Turnbull
Steven D'Aprano

A python bridge between versions

Mark Lawrence

tags

участники (14)