Mailman 3 March 2011 - Distutils-SIG

Buildout Sprint Report
by Jim Fulton March 18, 2011

March 18, 2011

We did some sprinting on buildout at PyCon. Alex Plugaru, fixed serveral bugs and clarified the development process a little. We had some good discussions about how buildout achieves isolation that I expect will lead to simplification in the future. I researched some test failures and ultimately had to disable some tests except on windows and newer linux kernels. The tests rely on being able to name a script on a shebang line. I worked on porting buildout to Python 3. My plan: 0. My goal is to have one code base that works in both Python 2 and Python 3. 1. Run 2to3 and get tests passing. I spent the last 4+ days working on this. There's a lot that 2to3 doesn't take care of. I think I'm getting pretty close and will be able to finish the work this weekend. Lennart Regebro provided valuable advice and moral support through this process. :) 2. Review the diffs and adjust to make buildout work under Python 2&3. Also look for stupidities committed in the heat of hacking. In particular, I used encode/decode calls in places as a band-aid to overcome mistakes in doing IO properly. 3. Make sure tests pass under both Python 2&3 and make a beta release. Note that I'm reliying on distribute because: - distutils2 aka packaging isn't ready (and when it is, it won't provide all of the features I need), - setuptools isn't ported to Python 3 yet. I'm sadly, but sorely, tempted to drop explicit support for setuptools. Maintaining support for both implementations is feeling like a waste of time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

1 0

Re: [Distutils] early preview of pythonv
by Carl Meyer March 17, 2011

March 17, 2011

Hi Vinay, I've certainly considered using PYTHONHOME. I know of at least two problems with the approach: 1 Having a dedicated virtualenv binary of some kind is more amenable to installing scripts with a shebang line that ensures they always run in the virtualenv. 2. Unless we change the semantics of PYTHONHOME, it would require the same kind of stdlib bootstrapping hacks that virtualenv currently uses. Avoiding this is a design goal. Carl Sent from my Android phone ----- Reply message ----- From: "Vinay Sajip" <vinay_sajip(a)yahoo.co.uk> Date: Wed, Mar 16, 2011 9:59 am Subject: [Distutils] early preview of pythonv To: <Distutils-Sig(a)Python.Org> Carl Meyer <carl <at> oddbird.net> writes: > This is an early prototype and will certainly require refinement (not to > mention most likely a PEP, at some point). Please try it out and let me > know if it works for you! I haven't tried it yet, but I'm curious: was an approach using PYTHONHOME considered? Or is PYTHONHOME off-limits for an application like this? Regards, Vinay Sajip _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org http://mail.python.org/mailman/listinfo/distutils-sig

1 0

reservations about pythonv
by Brandon Craig Rhodes March 16, 2011

March 16, 2011

I suspect that the current "pythonv" proposal should not become how virtualenv is implemented in the future. The proposal would add another exec() to every invocation of Python in the virtualenv. Not only is exec() among the most costly system calls on Unix systems, but it is even slower on Windows. Furthermore, the re-exec() will generate significant noise and annoyance - if not outright breakage - when people use binary tools like gdb and valgrind, because MYENV/bin/python will become a binary that looks nothing like the Python binary (none of the usual symbols, binary dependencies, and so forth) and that disappears almost immediately after being run and is replaced with a completely different binary. Do all popular binary debugging tools handle an exec() cleanly? At the very least everyone's debugging practices and scripts would have to be re- worked to ignore the stats and results that came back from that first ephemeral "pythonv" process image. I certainly agree with everyone that Python 3.3 should natively support a virtualenv mechanism, but I think that it should be the full "python" that gets copied. I realize that this means that the "python" binary will not get updates when the system Python does; but isn't the whole idea of a virtualenv that it insulates me from change in a way that using the system Python does not? I would be quite happy for either the "activate" script (or Python itself?) to issue a warning if the binary in the virtualenv has fallen behind the image in the system Python. -- Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon

3 2

Re: [Distutils] early preview of pythonv
by a.cavallo＠cavallinux.eu March 16, 2011

March 16, 2011

Hi, please have a look into my little project: http://pyvm.sf.net It does exactly this integration (and build integration and run time testing) on several different linux distros. It works stright out of the svn py27 trunk but the work flow can be implemented for 3.x. Regards, Antonio On Wed 16/03/11 13:00, "Jim Fulton" jim(a)zope.com wrote: > On Wed, Mar 16, 2011 at 4:31 AM, Carl Meyer <carl@oddbird > .net> wrote: > Hello all, > > > > Here at PyCon we've had some discussion about > building a > virtualenv-alike into Python core for Python 3.3. > The goal is to improve > on virtualenv by providing something that does what > virtualenv does > without requiring a copied Python binary, > symlinked/copied parts of the > standard library, or a forked site.py. > > > > (I'm not entirely sure that distutils-sig is the > right venue for > discussing this, but it's the closest I know of and > was recommended by > others in our conversations here; if there's a > better place please let > me know; maybe python-ideas? The patch as is stands > does affect > sysconfig.py, which is used more by distutils than > anything else.) > > > The idea we discussed is to add to Python's built-in > site.py the ability > to set paths up for a virtual environment, triggered > by certain > environment variables. Then, for convenience, > there'll be a small > executable which can be placed in the "bin/" > directory of a virtual > environment and knows how to set up these > environment variables and then > exec() the system Python binary. > > > > Larry Hastings had already created this wrapper > executable at last > year's PyCon, and tonight I made the necessary > modifications to site.py > and sysconfig.py to support it on the Python side. > The early prototype > is now working well (at least on Linux; I think it > ought to work on OS > X, and should work partially on Windows as well), > and I'd welcome review > and comment: https://bitbucket.org/carljm/cpythonv Look > in > Tools/pythonv/README.rst for > instructions. > > > This is an early prototype and will certainly > require refinement (not to > mention most likely a PEP, at some point). Please > try it out and let me > know if it works for you! > > Carl, > > Thanks for posting this. I'm very hopeful that buildout can use this > same mechanism to get the isolation it needs. I would really > appreciate it if buildout users who care about this would test it with > buildout. In particular, I know of 2 basic use cases: > > - Get complete isolation from local additions relative to the standard > Python distribution. > > - Have the ability to cherry pick some local additions while having > isolation from the rest. (Gary and Ubuntu, I'm looking at you.) > > Please, let's make sure this mechanism is enough. To raise the stakes > a bit, when this mechanism is available, presumably in Python 3.3, I > plan to use it exclusively to provide isolation in buildout. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://mail.python.org/mailman/listinfo/distutils-sig > >

3 2

Re: [Distutils] early preview of pythonv
by Carl Meyer March 16, 2011

March 16, 2011

Hi Antonio, Thanks for the pointer to this. It does look interesting, but I think it hits a rather different set of use cases than virtualenv or pythonv. In particular, virtualenv/pythonv is useful for situations where: - I may not have administrator access to the system, - I want to run many projects on exactly the same version of Python, but with different working sets of packages, - I'd like a cross-platform solution (including Windows and OS X) that maintains a similar workflow on all platforms. I may not be fully understanding pyvm, but it appears to me that it would not fit those use cases. So while I think it looks interesting and useful, I don't think its a viable alternative for the goals of pythonv. Carl Sent from my Android phone ----- Reply message ----- From: a.cavallo(a)cavallinux.eu Date: Wed, Mar 16, 2011 7:38 am Subject: [Distutils] early preview of pythonv To: "distutils-sig" <distutils-sig(a)python.org> Hi, please have a look into my little project: http://pyvm.sf.net It does exactly this integration (and build integration and run time testing) on several different linux distros. It works stright out of the svn py27 trunk but the work flow can be implemented for 3.x. Regards, Antonio On Wed 16/03/11 13:00, "Jim Fulton" jim(a)zope.com wrote: > On Wed, Mar 16, 2011 at 4:31 AM, Carl Meyer <carl@oddbird > .net> wrote: > Hello all, > > > > Here at PyCon we've had some discussion about > building a > virtualenv-alike into Python core for Python 3.3. > The goal is to improve > on virtualenv by providing something that does what > virtualenv does > without requiring a copied Python binary, > symlinked/copied parts of the > standard library, or a forked site.py. > > > > (I'm not entirely sure that distutils-sig is the > right venue for > discussing this, but it's the closest I know of and > was recommended by > others in our conversations here; if there's a > better place please let > me know; maybe python-ideas? The patch as is stands > does affect > sysconfig.py, which is used more by distutils than > anything else.) > > > The idea we discussed is to add to Python's built-in > site.py the ability > to set paths up for a virtual environment, triggered > by certain > environment variables. Then, for convenience, > there'll be a small > executable which can be placed in the "bin/" > directory of a virtual > environment and knows how to set up these > environment variables and then > exec() the system Python binary. > > > > Larry Hastings had already created this wrapper > executable at last > year's PyCon, and tonight I made the necessary > modifications to site.py > and sysconfig.py to support it on the Python side. > The early prototype > is now working well (at least on Linux; I think it > ought to work on OS > X, and should work partially on Windows as well), > and I'd welcome review > and comment: https://bitbucket.org/carljm/cpythonv Look > in > Tools/pythonv/README.rst for > instructions. > > > This is an early prototype and will certainly > require refinement (not to > mention most likely a PEP, at some point). Please > try it out and let me > know if it works for you! > > Carl, > > Thanks for posting this. I'm very hopeful that buildout can use this > same mechanism to get the isolation it needs. I would really > appreciate it if buildout users who care about this would test it with > buildout. In particular, I know of 2 basic use cases: > > - Get complete isolation from local additions relative to the standard > Python distribution. > > - Have the ability to cherry pick some local additions while having > isolation from the rest. (Gary and Ubuntu, I'm looking at you.) > > Please, let's make sure this mechanism is enough. To raise the stakes > a bit, when this mechanism is available, presumably in Python 3.3, I > plan to use it exclusively to provide isolation in buildout. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org http://mail.python.org/mailman/listinfo/distutils-sig

1 0

early preview of pythonv
by Carl Meyer March 16, 2011

March 16, 2011

Hello all, Here at PyCon we've had some discussion about building a virtualenv-alike into Python core for Python 3.3. The goal is to improve on virtualenv by providing something that does what virtualenv does without requiring a copied Python binary, symlinked/copied parts of the standard library, or a forked site.py. (I'm not entirely sure that distutils-sig is the right venue for discussing this, but it's the closest I know of and was recommended by others in our conversations here; if there's a better place please let me know; maybe python-ideas? The patch as is stands does affect sysconfig.py, which is used more by distutils than anything else.) The idea we discussed is to add to Python's built-in site.py the ability to set paths up for a virtual environment, triggered by certain environment variables. Then, for convenience, there'll be a small executable which can be placed in the "bin/" directory of a virtual environment and knows how to set up these environment variables and then exec() the system Python binary. Larry Hastings had already created this wrapper executable at last year's PyCon, and tonight I made the necessary modifications to site.py and sysconfig.py to support it on the Python side. The early prototype is now working well (at least on Linux; I think it ought to work on OS X, and should work partially on Windows as well), and I'd welcome review and comment: https://bitbucket.org/carljm/cpythonv Look in Tools/pythonv/README.rst for instructions. This is an early prototype and will certainly require refinement (not to mention most likely a PEP, at some point). Please try it out and let me know if it works for you! Thanks, Carl

3 4

Distribute and Python 3.2
by Vinay Sajip March 16, 2011

March 16, 2011

I'm working on Python 3 versions of pip and virtualenv, with the idea of keeping a common codebase for Python 2.x and 3.x so as to make maintenance easier. To date, I've made good progress: My port of pip (working with my port of virtualenv) passes 100 tests (2 are skipped) on Python 3.2 and 3.3a0. However, one roadblock is that distribute 0.6.14 doesn't work with Python 3.2 and later, because of the "abiflags" configuration parameter which is new in 3.2. Toshio Kuratomi has a fix for this checked in, and my tests have used a tarball of the distribute 0.6-maintenance tip with a name of distribute-0.6.15dev.tar.gz. It would be good if an official 0.6.15 release of distribute were made, incorporating that fix, now that 3.2 is out. Can someone tell me if there is a plan to release a 0.6.15, and if so, when it will be? Regards, Vinay Sajip P.S. IMO Toshio Kuratomi's fix could be better implemented as self.config_vars['abiflags'] = getattr(sys, 'abiflags', '') in the same block as all the other self.config_vars[...] assignments.

5 8

Distributes from pkg_resources.resource_string returns bytes under Python 3
by Lennart Regebro March 13, 2011

March 13, 2011

Distributes from pkg_resources.resource_string returns bytes under Python 3, which is pretty surprising. :-) Should we fix this? That would mean that we need to introduce a pkg_resources.resource_bytes that returns bytes under Python 3 and str (again) in Python 2. And probably we also need a pkg_resources.resource_unicode that returns Unicode under Python 2 and str under Python 3... Opinions on that?

2 2

Buildout documentation sprint at PyCon
by Jim Fulton March 13, 2011

March 13, 2011

I've started working on a buildout documentation project to provide decent sphinx documentation for buildout. This documentation will have the following structure: Introduction How-tos ... Special Topics ... Reference ... This is planned as a collection of relatively small relatively independent documents. Each document will be first and foremost **documentation**, however manuel will be used to try to assure that examples work. This effort lends itself to small contributions and seems well suited to sprint work if anyone is interested. This is a good opportunity to contribute to buildout and to learn about sphinx and manuel. I'll have the subversion project in place by the time the sprints start. I'll be there for all 4 sprint days. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

1 1

Re: [Distutils] [distribute] distribute_setup.py fails on Linux
by Ned Schumann March 13, 2011

March 13, 2011

Hello Arve, Running python2.6 distribute_setup.py works on a vanilla 64 bit CentOS 5 installation but fails on a default vanilla EC2 AMI Linux . Both Pythons were built with 'make altinstall'. Your note suggests that there was a problem with your Python installation. It appears that I too have an installation problem. I installed with 'make altinstall' but how could that go wrong? The python-2.6.6 regression tests for the EC2 version reveal ... test test_zipfile failed -- multiple errors occurred; run in verbose mode for details test_zipimport skipped -- No module named zlib On EC2, when python2.6 distribute_setup.py is run with a Pdb trace at line 1658 n tarfile.py ... (Pdb) print func(name, "r", fileobj, **kwargs) *** CompressionError: gzip module is not available Yet /usr/local/build/Python-2.6.6/Modules/zlib exists for the EC2 build. Please let me know a work-around. Is there anything I should suggest to AWS? With thanks, ned

1 0