Distribute without 2to3
I'm testing a branch of Python which provides out-of-the-box ability to create virtual enviroments à la virtualenv, and as part of that testing I have to install Distribute in newly created environments a lot. Though normally running 2to3 as part of the Distribute installation is not a big deal, for this testing I'm doing, it does slow things down a little. While waiting for some tests to finish, I thought I might as well look into whether Distribute could be made to run on 2.x and 3.x from a single code base, thereby avoiding the 2to3 step. I've made an attempt, and things seem to have gone reasonably smoothly: the conversion is done and all unit tests pass on 2.7, 3.2, 3.3 (my branch). The converted Distribute is at https://bitbucket.org/vinay.sajip/distribute3 In case any of you are interested at taking a look, I'd welcome any feedback you have. Regards, Vinay Sajip
On Tue, Jun 21, 2011 at 10:34, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I'm testing a branch of Python which provides out-of-the-box ability to create virtual enviroments à la virtualenv, and as part of that testing I have to install Distribute in newly created environments a lot. Though normally running 2to3 as part of the Distribute installation is not a big deal, for this testing I'm doing, it does slow things down a little.
While waiting for some tests to finish, I thought I might as well look into whether Distribute could be made to run on 2.x and 3.x from a single code base, thereby avoiding the 2to3 step.
I've made an attempt, and things seem to have gone reasonably smoothly: the conversion is done and all unit tests pass on 2.7, 3.2, 3.3 (my branch).
We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help. //Lennart
On 06/21/2011 11:21 AM, Lennart Regebro wrote:
On Tue, Jun 21, 2011 at 10:34, Vinay Sajip<vinay_sajip@yahoo.co.uk> wrote:
I'm testing a branch of Python which provides out-of-the-box ability to create virtual enviroments à la virtualenv, and as part of that testing I have to install Distribute in newly created environments a lot. Though normally running 2to3 as part of the Distribute installation is not a big deal, for this testing I'm doing, it does slow things down a little.
While waiting for some tests to finish, I thought I might as well look into whether Distribute could be made to run on 2.x and 3.x from a single code base, thereby avoiding the 2to3 step.
I've made an attempt, and things seem to have gone reasonably smoothly: the conversion is done and all unit tests pass on 2.7, 3.2, 3.3 (my branch). We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.
Afaik you can't catch exceptions in a way that is source-compatible with python 3.x and <2.6 due to the switch from "except <class>, <variable> to "except <class> as <variable>". Wichert.
On 21 June 2011 10:23, Wichert Akkerman <wichert@wiggy.net> wrote:
On 06/21/2011 11:21 AM, Lennart Regebro wrote:
On Tue, Jun 21, 2011 at 10:34, Vinay Sajip<vinay_sajip@yahoo.co.uk> wrote:
I'm testing a branch of Python which provides out-of-the-box ability to create virtual enviroments à la virtualenv, and as part of that testing I have to install Distribute in newly created environments a lot. Though normally running 2to3 as part of the Distribute installation is not a big deal, for this testing I'm doing, it does slow things down a little.
While waiting for some tests to finish, I thought I might as well look into whether Distribute could be made to run on 2.x and 3.x from a single code base, thereby avoiding the 2to3 step.
I've made an attempt, and things seem to have gone reasonably smoothly: the conversion is done and all unit tests pass on 2.7, 3.2, 3.3 (my branch).
We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.
Afaik you can't catch exceptions in a way that is source-compatible with python 3.x and <2.6 due to the switch from "except <class>, <variable> to "except <class> as <variable>".
You can, it just isn't particularly pretty - virtualenv and pip access the variable via sys.exc_info()[1], eg: except Exception: e = sys.exc_info()[1] Paul
Lennart Regebro <regebro <at> gmail.com> writes:
We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.
I'm not sure why 2.4 is a particular issue: I just tested on 2.4.6 without any failures. See https://gist.github.com/1037662 Of course, I'm assuming that "python setup.py test" gives adequate coverage. Regards, Vinay Sajip
On Tue, Jun 21, 2011 at 13:30, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Lennart Regebro <regebro <at> gmail.com> writes:
We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.
I'm not sure why 2.4 is a particular issue.
It isn't. The big difference is between 2.5 and 2.6. But each additional version brings it's own incompatibilities.
I just tested on 2.4.6 without any failures. See
That's positive but...
Of course, I'm assuming that "python setup.py test" gives adequate coverage.
It doesn't. Far from it, unfortunately. //Lennart
On 21 June 2011 13:42, Lennart Regebro <regebro@gmail.com> wrote:
On Tue, Jun 21, 2011 at 13:30, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Lennart Regebro <regebro <at> gmail.com> writes:
We still need to support Python 2.4, right? That's a trickier issue. But including six.py might help.
I'm not sure why 2.4 is a particular issue.
It isn't. The big difference is between 2.5 and 2.6. But each additional version brings it's own incompatibilities.
Really? In my experience dropping 2.4 support allows you to use the with statement (just as dropping 2.3 support allows you to use decorators) which is a big change. I've found the 2.5-2.6 changes to be much less dramatic. I have to jump through hoops a little bit to test context managers in a code base that remains 2.4 compatible (you can *write* context managers and remain 2.4 compatible - just not use them). Michael
I just tested on 2.4.6 without any failures. See
That's positive but...
Of course, I'm assuming that "python setup.py test" gives adequate coverage.
It doesn't. Far from it, unfortunately.
//Lennart _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Tue, Jun 21, 2011 at 15:03, Michael Foord <fuzzyman@gmail.com> wrote:
Really? In my experience dropping 2.4 support allows you to use the with statement (just as dropping 2.3 support allows you to use decorators) which is a big change.
Sure, but since we support 2.4 now, I don't think the code uses it. In my experience when making code run on both python 2 and python 3, the bets path is to run 2to3 on the code and make it run under Python 3, and then backport one version at a time. And in that case you aren't using any with-statements or context managers (because the code was running under 2.4 already before you ran 2to3 on it).
I've found the 2.5-2.6 changes to be much less dramatic.
Well, in 2.5 you have neither bytes literals, nor "except as", nor "from __future__ import print_function". All of these require big changes during a backport. Of course, you might have done them already for 2.7 even if they were not needed, but with 2.5 they *are* needed. //Lennart
Lennart Regebro <regebro <at> gmail.com> writes:
That's positive but...
Of course, I'm assuming that "python setup.py test" gives adequate coverage.
It doesn't. Far from it, unfortunately.
That's not a good place to be, given that we'll be reliant on Distribute for a while. Is there a more comprehensive test suite elsewhere? The stuff in the distribute tests folder (not setuptools/tests) looked out-of-date and, in places, broken. This version was developed by running 2to3 on the Distribute code to see what changes were needed, and then making the changes using a compatibility module using an approach similar to Benjamin Peterson's six library. It is of course possible that I've introduced some bugs during the conversion, but any that turn up should be very easy to fix. I'm using this Distribute version to try installing (via pysetup3) all allegedly Py-3K packages on PyPI. So far, out of around 400 packages where I could find a download URL for a source distribution, smoke testing revealed 190 apparent successes, 70 failures and 140 which were untested because of failure to get the tarball. A quick check of the failures shows that at least some of these are due to failures in the tarballs themselves, e.g. Markdown - setup.py is not Py3K compatible (contains print statements) ast2src - hardcoded to require Python 3.1 Benchmarker - failed to recognize --single-version-externally-managed cbucho - fails to import setuptools correctly in Py3K conditional code Of course some potential pysetup3 issues are showing up too, e.g. "cannot detect install method", which is one of the points of doing these tests. Regards, Vinay Sajip
On 06/21/2011 03:13 PM, Vinay Sajip wrote:
Benchmarker - failed to recognize --single-version-externally-managed --single-version-externally-managed is a setuptools option IIRC.
I've looked at its setup.py and found this: if arg1 == 'egg_info': from ez_setup import use_setuptools use_setuptools() if arg1 == 'bdist_egg': from setuptools import setup else: from distutils.core import setup So it is probably recognized as a setuptools project by pysteup3 (in packaging.util) but then uses bare distutils, which doesn't have the --single-version-managed option. Maybe in this case (when the call to setup.py --single--...) fails, we can fallback onto bare distutils install on packaging.install ? -- Alexis
On Tue, Jun 21, 2011 at 15:13, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I'm using this Distribute version to try installing (via pysetup3) all allegedly Py-3K packages on PyPI. So far, out of around 400 packages where I could find a download URL for a source distribution, smoke testing revealed 190 apparent successes, 70 failures and 140 which were untested because of failure to get the tarball. A quick check of the failures shows that at least some of these are due to failures in the tarballs themselves, e.g.
That's a good test. Next step is to try make a buildout with it, and then do the same under 2.6 and 2.4. If that all passes, it's in a good usable state, I would say. //Lennart
Lennart Regebro <regebro <at> gmail.com> writes:
That's a good test. Next step is to try make a buildout with it, and then do the same under 2.6 and 2.4. If that all passes, it's in a good usable state, I would say.
Thanks. I'm not familiar with buildout, so if someone more savvy wants to give it a quick try, be my guest :-) Regards, Vinay Sajip
Lennart Regebro <regebro <at> gmail.com> writes:
That's a good test. Next step is to try make a buildout with it, and then do the same under 2.6 and 2.4. If that all passes, it's in a good usable state, I would say.
I haven't had a chance to look at buildout and not sure what recipes need to be tried, but just testing installing the projects on PyPI using pythonv, pysetup3 and this Distribute version has been instructive. I was getting some errors with BitBucket's CDN serving up old versions of files, so I tweaked the version numbers on the Disrribute download archive (and fixed one or two bugs) and re-tested. Out of 398 packages on PyPI which have a Python 3 trove classifier, apparently 310 were installed without errors. The other 88 had errors, some of which are project related (e.g. 14 SyntaxErrors) or have specific version requirements (8 insist on Python 3.1, for example), and others of which are due to missing dependencies or missing README files (7 instances). I'm still looking, but I still haven't found any Distribute-related errors. Full results on the 88 failures are at https://gist.github.com/1037662 - IMO 310 out of 398 is not too shabby for this stage in the proceedings (around 78%). Regards, Vinay Sajip
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
Out of 398 packages on PyPI which have a Python 3 trove classifier, apparently 310 were installed without errors. The other 88 had errors, some of which are project related (e.g. 14 SyntaxErrors) or have specific version requirements (8 insist on Python 3.1, for example), and others of which are due to missing dependencies or missing README files (7 instances).
During this testing I came across a couple of pysetup3 issues: 1. It doesn't appear to like archive names containing spaces (example: the "ISO8583 Module" on PyPI) 2. It ("pysetup3 install tarball") leaves temporary folders lying around (the ones that contain the expanded archive to be installed, e.g. /tmp tmpiv3p7u pythonselect-1.2 The contents of pythonselect-1.2.tar.gz Although there are similar issues on the tracker, they don't seem to be quite the same - are these known issues, or should I log them? Regards, Vinay Sajip
Hmm. The "option --single-version-externally-managed not recognized" errors look like they might be a distribute bug. The syntax errors look like they are syntax errors in the setup.py files and not distribute bugs. //Lennart On Wed, Jun 22, 2011 at 01:46, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Lennart Regebro <regebro <at> gmail.com> writes:
That's a good test. Next step is to try make a buildout with it, and then do the same under 2.6 and 2.4. If that all passes, it's in a good usable state, I would say.
I haven't had a chance to look at buildout and not sure what recipes need to be tried, but just testing installing the projects on PyPI using pythonv, pysetup3 and this Distribute version has been instructive. I was getting some errors with BitBucket's CDN serving up old versions of files, so I tweaked the version numbers on the Disrribute download archive (and fixed one or two bugs) and re-tested.
Out of 398 packages on PyPI which have a Python 3 trove classifier, apparently 310 were installed without errors. The other 88 had errors, some of which are project related (e.g. 14 SyntaxErrors) or have specific version requirements (8 insist on Python 3.1, for example), and others of which are due to missing dependencies or missing README files (7 instances).
I'm still looking, but I still haven't found any Distribute-related errors. Full results on the 88 failures are at https://gist.github.com/1037662 - IMO 310 out of 398 is not too shabby for this stage in the proceedings (around 78%).
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Lennart Regebro <regebro <at> gmail.com> writes:
The "option --single-version-externally-managed not recognized" errors look like they might be a distribute bug.
Or perhaps a pysetup3 bug - I think Alexis commented on this.
The syntax errors look like they are syntax errors in the setup.py files and not distribute bugs.
Yes, I characterised those as project-related (meaning the PyPI project being installed), as are the errors where setup.py tries to read a missing readme.txt or similar file. Regards, Vinay Sajip
participants (6)
-
Alexis Métaireau -
Lennart Regebro -
Michael Foord -
Paul Nasrat -
Vinay Sajip -
Wichert Akkerman