On Tue, Jun 21, 2011 at 10:34, Vinay Sajip wrote:

I'm testing a branch of Python which provides out-of-the-box ability to create virtual enviroments à la virtualenv, and as part of that testing I have to install Distribute in newly created environments a lot. Though normally running 2to3 as part of the Distribute installation is not a big deal, for this testing I'm doing, it does slow things down a little.

While waiting for some tests to finish, I thought I might as well look into whether Distribute could be made to run on 2.x and 3.x from a single code base, thereby avoiding the 2to3 step.

I've made an attempt, and things seem to have gone reasonably smoothly: the conversion is done and all unit tests pass on 2.7, 3.2, 3.3 (my branch).

We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help. //Lennart

On 21 June 2011 10:23, Wichert Akkerman wrote:

On 06/21/2011 11:21 AM, Lennart Regebro wrote:

...

On Tue, Jun 21, 2011 at 10:34, Vinay Sajip  wrote:

...

I'm testing a branch of Python which provides out-of-the-box ability to create virtual enviroments à la virtualenv, and as part of that testing I have to install Distribute in newly created environments a lot. Though normally running 2to3 as part of the Distribute installation is not a big deal, for this testing I'm doing, it does slow things down a little.

While waiting for some tests to finish, I thought I might as well look into whether Distribute could be made to run on 2.x and 3.x from a single code base, thereby avoiding the 2to3 step.

I've made an attempt, and things seem to have gone reasonably smoothly: the conversion is done and all unit tests pass on 2.7, 3.2, 3.3 (my branch).

We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.

Afaik you can't catch exceptions in a way that is source-compatible with python 3.x and <2.6 due to the switch from "except <class>, <variable> to "except <class> as <variable>".

You can, it just isn't particularly pretty - virtualenv and pip access the variable via  sys.exc_info()[1], eg: except Exception: e = sys.exc_info()[1] Paul

On Tue, Jun 21, 2011 at 13:30, Vinay Sajip wrote:

Lennart Regebro writes:

...

We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.

I'm not sure why 2.4 is a particular issue.

It isn't. The big difference is between 2.5 and 2.6. But each additional version brings it's own incompatibilities.

I just tested on 2.4.6 without any failures. See

https://gist.github.com/1037662

That's positive but...

Of course, I'm assuming that "python setup.py test" gives adequate coverage.

It doesn't. Far from it, unfortunately. //Lennart

Lennart Regebro writes:

That's positive but...

...

Of course, I'm assuming that "python setup.py test" gives adequate coverage.

It doesn't. Far from it, unfortunately.

That's not a good place to be, given that we'll be reliant on Distribute for a while. Is there a more comprehensive test suite elsewhere? The stuff in the distribute tests folder (not setuptools/tests) looked out-of-date and, in places, broken. This version was developed by running 2to3 on the Distribute code to see what changes were needed, and then making the changes using a compatibility module using an approach similar to Benjamin Peterson's six library. It is of course possible that I've introduced some bugs during the conversion, but any that turn up should be very easy to fix. I'm using this Distribute version to try installing (via pysetup3) all allegedly Py-3K packages on PyPI. So far, out of around 400 packages where I could find a download URL for a source distribution, smoke testing revealed 190 apparent successes, 70 failures and 140 which were untested because of failure to get the tarball. A quick check of the failures shows that at least some of these are due to failures in the tarballs themselves, e.g. Markdown - setup.py is not Py3K compatible (contains print statements) ast2src - hardcoded to require Python 3.1 Benchmarker - failed to recognize --single-version-externally-managed cbucho - fails to import setuptools correctly in Py3K conditional code Of course some potential pysetup3 issues are showing up too, e.g. "cannot detect install method", which is one of the points of doing these tests. Regards, Vinay Sajip

On Tue, Jun 21, 2011 at 15:03, Michael Foord wrote:

Really? In my experience dropping 2.4 support allows you to use the with statement (just as dropping 2.3 support allows you to use decorators) which is a big change.

Sure, but since we support 2.4 now, I don't think the code uses it. In my experience when making code run on both python 2 and python 3, the bets path is to run 2to3 on the code and make it run under Python 3, and then backport one version at a time. And in that case you aren't using any with-statements or context managers (because the code was running under 2.4 already before you ran 2to3 on it).

I've found the 2.5-2.6 changes to be much less dramatic.

Well, in 2.5 you have neither bytes literals, nor "except as", nor "from __future__ import print_function". All of these require big changes during a backport. Of course, you might have done them already for 2.7 even if they were not needed, but with 2.5 they *are* needed. //Lennart

Lennart Regebro writes:

That's a good test. Next step is to try make a buildout with it, and then do the same under 2.6 and 2.4. If that all passes, it's in a good usable state, I would say.

Thanks. I'm not familiar with buildout, so if someone more savvy wants to give it a quick try, be my guest :-) Regards, Vinay Sajip

Vinay Sajip writes:

Out of 398 packages on PyPI which have a Python 3 trove classifier, apparently 310 were installed without errors. The other 88 had errors, some of which are project related (e.g. 14 SyntaxErrors) or have specific version requirements (8 insist on Python 3.1, for example), and others of which are due to missing dependencies or missing README files (7 instances).

During this testing I came across a couple of pysetup3 issues: 1. It doesn't appear to like archive names containing spaces (example: the "ISO8583 Module" on PyPI) 2. It ("pysetup3 install tarball") leaves temporary folders lying around (the ones that contain the expanded archive to be installed, e.g. /tmp tmpiv3p7u pythonselect-1.2 The contents of pythonselect-1.2.tar.gz Although there are similar issues on the tracker, they don't seem to be quite the same - are these known issues, or should I log them? Regards, Vinay Sajip

Lennart Regebro writes:

The "option --single-version-externally-managed not recognized" errors look like they might be a distribute bug.

Or perhaps a pysetup3 bug - I think Alexis commented on this.

The syntax errors look like they are syntax errors in the setup.py files and not distribute bugs.

Yes, I characterised those as project-related (meaning the PyPI project being installed), as are the errors where setup.py tries to read a missing readme.txt or similar file. Regards, Vinay Sajip