2011/11/21 Terry Reedy

I strongly recommend that where it makes a difference, the pypy python3 project target 3.3. In particular, don't reproduce the buggy narrow-build behavior of 3.2 and before (perhaps pypy avoids this already). Do include the new unicode capi in cpyext. I anticipate that 3.3 will see more production use than 3.2

In the current 2.7-compatible version, PyPy already uses wchar_t for its Unicode string, i.e. it is always a wide build with gcc and a narrow build on Windows. But this will certainly change for the 3.x port. PyPy already supports different internal representations for the same visible user type, and it makes sense to have 1-byte, 2-bytes and 4-bytes unicode types and try to choose the most efficient representation. As for the C API... getting a pointer out of a PyPy string already requires to allocate and fill a new non-movable buffer (since all memory allocated by PyPy is movable). So cpyext could support the new API for sure, but it's unlikely to give any performance benefit to an extension module. -- Amaury Forgeot d'Arc

Giampaolo Rodolà, 22.11.2011 10:21:

2011/11/21 Terry Reedy:

...

I strongly recommend that where it makes a difference, the pypy python3 project target 3.3. In particular, don't reproduce the buggy narrow-build behavior of 3.2 and before (perhaps pypy avoids this already). Do include the new unicode capi in cpyext. I anticipate that 3.3 will see more production use than 3.2

Is there a reason in particular?

Well, Py3 still has a lot to catch up in terms of wide spread distribution compared to Py2.x, and new users will usually start using the most up to date release, which will soon be 3.3. Besides, 3.3 has received various optimisations that make it more suitable for production use than 3.2, including the above mentioned Unicode optimisations. Stefan

Maciej Fijalkowski, 22.11.2011 15:46:

On Tue, Nov 22, 2011 at 3:15 PM, Stefan Behnel wrote:

...

Giampaolo Rodolà, 22.11.2011 10:21:

...

2011/11/21 Terry Reedy:

...

I strongly recommend that where it makes a difference, the pypy python3 project target 3.3. In particular, don't reproduce the buggy narrow-build behavior of 3.2 and before (perhaps pypy avoids this already). Do include the new unicode capi in cpyext. I anticipate that 3.3 will see more production use than 3.2

Is there a reason in particular?

Well, Py3 still has a lot to catch up in terms of wide spread distribution compared to Py2.x, and new users will usually start using the most up to date release, which will soon be 3.3.

Besides, 3.3 has received various optimisations that make it more suitable for production use than 3.2, including the above mentioned Unicode optimisations.

Note that I was referring to Terry's "more production use" comment here, not to the "PyPy should target 3.3 instead of 3.2" part.

PyPy's py3k branch targets Python 3.2 until 3.3 is released and very likely 3.3 afterwards. Optimizations are irrelevant really in the case of PyPy.

I admit that I wasn't very clear in my wording. As Terry pointed out, the Unicode changes in Py3.3 are not only for speed and/or memory performance improvements, they also improve the compliance of the Unicode implementation, which implies a behavioural change. Since PyPy appears to have implemented the current wide/narrow behaviour of Py2 and Py3.[012] already, I see no reason not to target 3.2 for the time being as it does not require substantial changes in this part. Compliance with Py3.3 will then require implementing the new behaviour. Stefan

2011/11/22 Philip Jenvey

One reason to target 3.2 for now is it's not a moving target. There's overhead involved in managing modifications to the pure python standard lib needed for PyPy, tracking 3.3 changes as they happen as well exacerbates this.

The plans to split the standard lib into its own repo separate from core CPython will of course help alternative implementations here.

I don't see how it would help here. Copying the CPython Lib/ directory is not difficult, even though PyPy made slight modifications to the files, and even without any merge tool. OTOH when PyPy changed minor versions (from 2.7.0 to 2.7.2 IIRC) most of the work was to follow the various tiny fixes made to the built-in modules: _io, _ssl and _multiprocessing. -- Amaury Forgeot d'Arc

On Wed, Nov 23, 2011 at 11:13 PM, Philip Jenvey wrote:

On Nov 22, 2011, at 12:43 PM, Amaury Forgeot d'Arc wrote:

...

2011/11/22 Philip Jenvey One reason to target 3.2 for now is it's not a moving target. There's overhead involved in managing modifications to the pure python standard lib needed for PyPy, tracking 3.3 changes as they happen as well exacerbates this.

The plans to split the standard lib into its own repo separate from core CPython will of course help alternative implementations here.

I don't see how it would help here. Copying the CPython Lib/ directory is not difficult, even though PyPy made slight modifications to the files, and even without any merge tool.

Pulling in a separate stdlib as a subrepo under the PyPy repo would certainly make this whole process easier.

But you're right, if we track CPython's default branch (3.3) we can make many if not all of the PyPy modifications upstream   (until the 3.3rc1 code freeze) instead of in PyPy's modified-3.x directory. Maintaining that modified-3.x dir after every resync can be tedious.

-- Philip Jenvey

The problem is not with maintaining the modified directory. The problem was always things like changing interface between the C version and the Python version or introduction of new stuff that does not run on pypy because it relies on refcounting. I don't see how having a subrepo helps here.

On Thu, Nov 24, 2011 at 07:46, Nick Coghlan wrote:

On Thu, Nov 24, 2011 at 10:20 PM, Maciej Fijalkowski wrote:

...

The problem is not with maintaining the modified directory. The problem was always things like changing interface between the C version and the Python version or introduction of new stuff that does not run on pypy because it relies on refcounting. I don't see how having a subrepo helps here.

Indeed, the main thing that can help on this front is to get more modules to the same state as heapq, io, datetime (and perhaps a few others that have slipped my mind) where the CPython repo actually contains both C and Python implementations and the test suite exercises both to make sure their interfaces remain suitably consistent (even though, during normal operation, CPython users will only ever hit the C accelerated version).

This not only helps other implementations (by keeping a Python version of the module continuously up to date with any semantic changes), but can help people that are porting CPython to new platforms: the C extension modules are far more likely to break in that situation than the pure Python equivalents, and a relatively slow fallback is often going to be better than no fallback at all. (Note that ctypes based pure Python modules *aren't* particularly useful for this purpose, though - due to the libffi dependency, ctypes is one of the extension modules most likely to break when porting).

And the other reason I plan to see this through before I die is to help distribute the maintenance burden. Why should multiple VMs fix bad assumptions made by CPython in their own siloed repos and then we hope the change gets pushed upstream to CPython when it could be fixed once in a single repo that everyone works off of?

On Fri, Nov 25, 2011 at 12:37, Antoine Pitrou wrote:

On Fri, 25 Nov 2011 12:37:59 -0500 Brett Cannon wrote:

...

On Thu, Nov 24, 2011 at 07:46, Nick Coghlan wrote:

...

On Thu, Nov 24, 2011 at 10:20 PM, Maciej Fijalkowski

...

wrote:

...

The problem is not with maintaining the modified directory. The problem was always things like changing interface between the C version and the Python version or introduction of new stuff that does not run on pypy because it relies on refcounting. I don't see how having a subrepo helps here.

Indeed, the main thing that can help on this front is to get more modules to the same state as heapq, io, datetime (and perhaps a few others that have slipped my mind) where the CPython repo actually contains both C and Python implementations and the test suite exercises both to make sure their interfaces remain suitably consistent (even though, during normal operation, CPython users will only ever hit the C accelerated version).

This not only helps other implementations (by keeping a Python version of the module continuously up to date with any semantic changes), but can help people that are porting CPython to new platforms: the C extension modules are far more likely to break in that situation than the pure Python equivalents, and a relatively slow fallback is often going to be better than no fallback at all. (Note that ctypes based pure Python modules *aren't* particularly useful for this purpose, though - due to the libffi dependency, ctypes is one of the extension modules most likely to break when porting).

And the other reason I plan to see this through before I die

Uh! Any bad news? :/

Sorry, turn of phrase in English which didn't translate well. I just meant "when I get to it, which could quite possibly be a *long* time from now". This year has been absolutely insane for me personally (if people care, the details are shared on Google+ or you can just ask me), so I am just not promising anything for Python on a short timescale (although I'm still hoping the final details for bootstrapping importlib won't be difficult to work out so I can meet a personal deadline of PyCon).

Le 25/11/2011 19:21, Amaury Forgeot d'Arc a écrit :

And oh, I almost forgot distutils, which needs to parse some Makefile which of course does not exist in PyPy.

This is a bug (#10764) that I intend to fix for the next releases of 2.7 and 3.2. I also want to fix all modules that use sys.version[:2] to get 'X.Y', which is a CPython implementation detail. I find PyPy and excellent project, so you can send any bugs in distutils, sysconfig, site and friends my way! I also hope to make distutils2 compatible with PyPy before 2012. Cheers

2011/11/22 Terry Reedy

On 11/22/2011 3:28 PM, Philip Jenvey wrote:

One reason to target 3.2 for now is it's not a moving target.

...

Neither is the basic design and behavior of the new unicode implementation. On 3.2 narrow builds, including Windows

...

...

...

len('\U00010101') 2

With 3.3, the answer will be, properly, 1. I suspect that becoming compatible with that, and all that it implies for many other examples, will be the biggest hurdle for PyPy becoming compatible with 3.3.

PyPy currently defines unicode as arrays of wchar_t, so only Windows uses a narrow unicode build. It will probably change though, and for performance reasons it makes indeed sense to have different internal representations for the same external type. PyPy already does this for several types (there is a special version of dict specialized for string keys, and the 2.7 range() returns a list that does not need to allocate its items, and can turn into a "real" list as soon as you modify it), so I would not qualify this task as a big hurdle, compared to other optimizations done in similar areas. -- Amaury Forgeot d'Arc

On Nov 22, 2011, at 02:15 PM, Stefan Behnel wrote:

Well, Py3 still has a lot to catch up in terms of wide spread distribution compared to Py2.x, and new users will usually start using the most up to date release, which will soon be 3.3.

Besides, 3.3 has received various optimisations that make it more suitable for production use than 3.2, including the above mentioned Unicode optimisations.

3.3 won't be released (according to PEP 398's current schedule) until August of next year. I think that's too long to wait before pushing for widespread adoption of Python 3. Hopefully, we're going to be making a dent in that in the next version of Ubuntu. We're actively starting to port a handful of desktop applications (including their dependency stacks) to Python 3.2, which I think is a fine release for doing so. I owe a blog post about this, but please do contact me if you want to get involved. Cheers, -Barry

On 2011-11-22, at 17:41 , Stephen J. Turnbull wrote:

Barry Warsaw writes:

...

Hopefully, we're going to be making a dent in that in the next version of Ubuntu.

This is still a big mess in Gentoo and MacPorts, though. MacPorts hasn't done anything about ceating a transition infrastructure AFAICT. What kind of "transition infrastructure" would it need? It's definitely not going to replace the Apple-provided Python out of the box, so setting `python` to a python3 is not going to happen.

Gentoo has its "eselect python set VERSION" stuff, but it's very dangerous to set to a Python 3 version, as many things go permanently wonky once you do. (So far I've been able to work around problems this creates, but it's not much fun.) Macports provide `port select` which I believe has the same function (you need to install the `python_select` for it to be configured for

It doesn't define a `python3`, so maybe that? Is there a document somewhere on what kind of things distros need for a transition plan? the Python group), the syntax is port `select --set python $VERSION`:

python --version Python 2.6.1 sudo port select --set python python32 Selecting 'python32' for 'python' succeeded. 'python32' is now active. python --version Python 3.2.2

Xavier Morel writes:

On 2011-11-22, at 17:41 , Stephen J. Turnbull wrote:

...

Barry Warsaw writes:

...

...

Hopefully, we're going to be making a dent in that in the next version of Ubuntu.

...

This is still a big mess in Gentoo and MacPorts, though. MacPorts hasn't done anything about ceating a transition infrastructure AFAICT.

What kind of "transition infrastructure" would it need? It's definitely not going to replace the Apple-provided Python out of the box, so setting `python` to a python3 is not going to happen.

Sure, but many things do shadow Apple-provided software if you set PATH=/opt/local/bin:$PATH. I'm not sure what infrastructure is required, but I can't really see MacPorts volunteers doing a 100% conversion the way that Ubuntu's paid developers can. So there will be a long transition period, and I wouldn't be surprised if multiple versions of Python 2 and multiple versions of Python 3 will typically need to be simultaneously available to different ports.

It doesn't define a `python3`, so maybe that?

A python3 symlink or script would help a little bit, but I don't think that's necessary or sufficient, because ports already can and do depend on Python x.y, not just Python x.

Is there a document somewhere on what kind of things distros need for a transition plan?

I'm hoping Barry's blog will be a good start.<wink/>

Macports provide `port select` which I believe has the same function (you need to install the `python_select` for it to be configured for the Python group), the syntax is port `select --set python $VERSION`:

Sure. I haven't had the nerve to do this on MacPorts because "port" is such a flaky thing (not so much port itself, but so many ports assume that the port maintainer's local configuration is what others' systems use, so I stay as vanilla as possible -- I rather doubt that many ports are ready for Python 3, and I'm not willing to be a guinea pig). The problem that I've run into with Gentoo is that *even when the ebuild is prepared for Python 3* assumptions about the Python current when the ebuild is installed/upgraded gets baked into the installation (eg, print statement vs. print function), but some of the support scripts just call "python" or something like that. OTOH, a few ebuilds don't support Python 3 (or in a ebuild that nominally supports Python 3, upstream does something perfectly reasonable for Python 2 like assume that Latin-1 characters are acceptable in a ChangeLog, and the ebuild maintainer doesn't test under Python 3 so it slips through) so I have to do an eselect dance while emerging ... and in the meantime things that expect Python 3 as the system Python break. So far, in Gentoo I've always been able to wiggle out of such problems by doing the eselect dance two or three times with the ebuild that is outdated, and then a couple of principal prerequisites or dependencies at most. Given my experience with MacPorts I *very much* expect similar issues with its ports.

Ned Deily writes:

In article <87fwhfqywr.fsf@uwakimon.sk.tsukuba.ac.jp>, "Stephen J. Turnbull" wrote:

...

I haven't had the nerve to do this on MacPorts because "port" is such a flaky thing (not so much port itself, but so many ports assume that the port maintainer's local configuration is what others' systems use, so I stay as vanilla as possible -- I rather doubt that many ports are ready for Python 3, and I'm not willing to be a guinea pig).

I think your fears are unfounded. MacPort's individual port files are supposed to be totally independent of the setting of 'port select'.

If you think I'm complaining or imagining things, you're missing the point. My fears are *not* unfounded. For personal use, I wanted Python 2.6 to be default using "port select", and things went wonky. Some things just didn't work, or disappeared. Reverting to 2.5 fixed, so I left it that way for a while. I tried it again with Python 2.7, same deal, different ports. Maybe those would have been considered bugs in "port select", I don't know. But reverting was easy, "fixed" things, and I won't try it with Python 3 (until I have a sacrificial system available). Also, the MacPorts solution is very resource intensive for users: I have *seven* Python stacks on the Mac where I'm typing this -- the only version of Python I've been able to eliminate once it has been installed so far is 3.0! although I could probably get rid of 3.1 now). It also leads to fragmentation (*all* of my 2.x stacks are incomplete, I can't do without any of them), and a couple of extra frustrating steps in finding the code that raised an exceptions or whatever. Not to mention that it's in my face daily: "port outdated" frequently lines up 3, occasionally 4 versions of the same port. This *only* happens with Python! And there's no way that many ports are ready for Python 3, because their upstreams aren't! I think that projects that would like to move to Python 3 are going to find they get pushback from Mac users who "don't need" *yet another* Python stack installed. Note that Gentoo has globally switched off the python USE flag[1] (I suspect that the issue is that one-time configuration utilities can pull in a whole Python stack that mostly duplicates Python content required for Gentoo to work at all).

Also right now besides the Python port group transition, the project has been swamped with issues arising from the Xcode 4 introduction for Lion, mandating the transition from gcc to clang or llvm-gcc.

Sure, I understand that kind of thing. That doesn't mean it improves the user experience with Python, especially Python 3. It helps if you can get widespread adoption at similar pace across the board rather than uneven diffusion with a few niches moving really fast. It's like Lao Tse didn't quite say: the most successful leaders are those who hustle and get a few steps ahead of the crowd wherever it's heading. But you need a crowd moving in the same direction to execute that strategy! So I'd like see people who *already* have the credibility with their distros to advocate Python 3. If Ubuntu's going to lead, now's a good time to join them. (Other things being equal, of course -- but then, other things are never equal, so it may as well be now anyway.<wink>) If that doesn't happen, well, Python and Python 3 will survive. But I'd rather to see them dominate. Footnotes: [1] According to the notes for the ibus ebuild.

On Wed, Nov 23, 2011 at 01:41:46AM +0900, Stephen J. Turnbull wrote:

Barry Warsaw writes:

...

Hopefully, we're going to be making a dent in that in the next version of Ubuntu.

This is still a big mess in Gentoo and MacPorts, though. MacPorts hasn't done anything about ceating a transition infrastructure AFAICT. Gentoo has its "eselect python set VERSION" stuff, but it's very dangerous to set to a Python 3 version, as many things go permanently wonky once you do. (So far I've been able to work around problems this creates, but it's not much fun.) I have no experience with this in Debian, Red Hat (and derivatives) or *BSD, but I have to suspect they're no better. (Well, maybe Red Hat has learned from its 1.5.2 experience! :-)

For Fedora (and currently, Red Hat is based on Fedora -- a little more about that later, though), we have parallel python2 and python3 stacks. As time goes on we've slowly brought more python-3 compatible modules onto the python3 stack (I believe someone had the goal a year and a half ago to get a complete pylons web development stack running on python3 on Fedora which brought a lot of packages forward). Unlike Barry's work with Ubuntu, though, we're mostly chiselling around the edges; we're working at the level where there's a module that someone needs to run something (or run some optional features of something) that runs on python3.

I don't have any connections to the distros, so can't really offer to help directly. I think it might be a good idea for users to lobby (politely!) their distros to work on the transition.

Where distros aren't working on parallel stacks, there definitely needs to be some transition plan. With my experience with parallel stacks, the best help there is to 1) help upstreams port to py3k (If someone can get PIL's py3k support finished and into a released package, that would free up a few things). 2) open bugs or help with creating python3 packages of modules when the upstream support is there. Depending on what software Barry's talking about porting to python3, that could be a big incentive as well. Just like with the push in Fedora to have pylons run on python3, I think that having certain applications that run on python3 and therefore need to have stacks of modules that support it is one of the prime ways that distros become motivated to provide python3 packages and support. This is basically the "killer app" idea in a new venue :-) -Toshio

On Nov 22, 2011, at 09:13 AM, Toshio Kuratomi wrote:

For Fedora (and currently, Red Hat is based on Fedora -- a little more about that later, though), we have parallel python2 and python3 stacks.

Debian (and thus Ubuntu) also has separate Python 2 and 3 stacks. In general, if you have a Python package (e.g. on PyPI) called 'foo', you'll have a Debian binary package called python-foo for the Python 2 version, and python3-foo for the Python 3 version. /usr/bin/python will always (modulo perhaps PEP 394) point to Python 2.x with Python 3 accessible via /usr/bin/python3. The minor version numbered Python binaries are also available. Debian's infrastructure makes it fairly easy to support multiple versions of Python at the same time, and of course to support both a Python 2 and 3 stack simultaneously. It's also fairly easy to switch the default Python version. Binary packages containing pure-Python are space efficient, sharing one copy of the Python source code for all supported versions. A symlink farm is used to manage the incompatibilities in .pyc files, but only for Python 2, since PEPs 3147 and 3149 solve this problem in a better way for Python 3 (no symlink farms necessary). The one additional complication though is that extension modules must be built for each supported version, and all .so's are included in a single binary package. E.g. if python-foo has an extension module, it will contain the 2.6 .so and the 2.7 .so. For the next version of Ubuntu, we will be dropping Python 2.6 support, so our binary packages are rebuilt to contain only the 2.7 version of the extension module. Details on how Debian packages Python, including its deviations from upstream, are available here: http://wiki.debian.org/Python Ubuntu's deviations from Debian and other details are available here: https://wiki.ubuntu.com/Python

Unlike Barry's work with Ubuntu, though, we're mostly chiselling around the edges; we're working at the level where there's a module that someone needs to run something (or run some optional features of something) that runs on python3.

This is great, because it means Fedora's taking kind of a bottom-up approach, while Ubuntu is taking a more top-down approach. Working together, we'll change the world. :) The key here is that we push as many of the changes as possible as far upstream as possible. I know Toshio and David agree, we want to get upstream package authors and application developers to support Python 3 as much as possible. I hope there will be no cases where a distro has to fork a package or application to support Python 3, although we will do it if there's no other way. Most likely for Ubuntu though, that would be pushing the changes into Debian.

Where distros aren't working on parallel stacks, there definitely needs to be some transition plan. With my experience with parallel stacks, the best help there is to 1) help upstreams port to py3k (If someone can get PIL's py3k support finished and into a released package, that would free up a few things). 2) open bugs or help with creating python3 packages of modules when the upstream support is there.

+1

Depending on what software Barry's talking about porting to python3, that could be a big incentive as well. Just like with the push in Fedora to have pylons run on python3, I think that having certain applications that run on python3 and therefore need to have stacks of modules that support it is one of the prime ways that distros become motivated to provide python3 packages and support. This is basically the "killer app" idea in a new venue :-)

Again, wholehearted +1. For now, we are not spending much time on server applications, though I've seen promising talk about Twisted ports to Python 3. We're looking specifically at desktop applications, such as Update Manager, Software Center, Computer Janitor, etc. Those may be fairly Ubuntu and/or Debian specific, but the actual applications themselves aren't too difficult to port. E.g. switching to Gnome object introspection, which already supports Python 3. We can easily identify the dependency stack for the desktop applications we're targeting, which leads us to looking at ports of the dependent libraries, and that benefits all Python users. Our goal is for the Ubuntu 14.04 LTS release (in April 2014) to have no Python 2 on the release images, or in our "main" archive, so everything you'd get on your desktop in a default install would be Python 3. For the upcoming 12.04 LTS release, I'd be happy if we had just a couple of Python 3 applications on the desktop by default. I see the work going on in Fedora/RedHat, Debian/Ubuntu, and other distributions as applying some positive momentum on pushing the Python community over the tipping point for Python 3 support. Cheers, -Barry

Dirkjan Ochtman writes:

On Tue, Nov 22, 2011 at 17:41, Stephen J. Turnbull wrote:

...

This is still a big mess in Gentoo and MacPorts, though.  MacPorts hasn't done anything about ceating a transition infrastructure AFAICT. Gentoo has its "eselect python set VERSION" stuff, but it's very dangerous to set to a Python 3 version, as many things go permanently wonky once you do.  (So far I've been able to work around problems this creates, but it's not much fun.)

Problems like what?

Like those I explained later in the post, which you cut. But I'll repeat. Some ebuilds are not prepared for Python 3, so must be emerged with a Python 2 eselected (and sometimes they need a specific Python 2). Some which are prepared don't get linted often enough, so new ebuilds are DOA because of an accented character in a changelog triggering a Unicode exception or similar dumb things like that.

...

I don't have any connections to the distros, so can't really offer to help directly.  I think it might be a good idea for users to lobby (politely!)  their distros to work on the transition.

Please create a connection to your distro by filing bugs as you encounter them?

No, thank you. File bugs, maybe, although most of the bugs I encounter in Gentoo are already in the database (often with multiple regressions going back a year or more), I could do a little more of that. (Response in the past has not been encouraging.) But I don't have time for distro politics. Is lack of Python 3-readiness considered a bug by Gentoo?

Dirkjan Ochtman writes:

I'm sorry for the lack of response in the past. I looked at Gentoo's Bugzilla and didn't find any related bugs you reported or were CC'ed on, can you name some of them?

This isn't about my bugs; I've been able to work through them satisfactorily. It's about what I perceive as a need for simultaneous improvement in Python 3 support across several distros, covering enough users to establish momentum. I don't think Python 3 needs to (or even can) replace Python 2 as the system python in the near future. But the "python" that is visible to users (at least on single-user systems) should be choosable by the user. eselect (on Gentoo) and port select (on MacPorts) *appear* to provide this, but it doesn't work very well.

...

Is lack of Python 3-readiness considered a bug by Gentoo?

Definitely. Again, we are trying to hard to make things better, but there's a lot to do and going through version bumps sometimes wins out over addressing the hard problems.

Well, as I see it the two hard problems are (1) the stack-per-python- minor-version solution is ugly and unattractive to users, and (2) the "select python" utility needs to be safe. This probably means that python-using ebuilds need to complain if the Python they find isn't a Python 2 version by default.