[Distutils] zc.buildout and site-packages

[Kevin Teague]

On Oct 22, 2009, at 10:43 AM, Tres Seaver wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Aspeli wrote:
>> Hi,
>>
>> Is there a way (apart from putting buildout in a virtualenv with -- 
>> no-site-
>> packages) to tell buildout *not* to put site-packages as the first  
>> line in the
>> mangled sys.path when it generates scripts?
>>
>> We have people doing horrid things to their global python, and we  
>> need the
>> buildout to be safe and isolated in these environments.
>
> Using a --no-site-packages virtualenv to drive the buildout is a  
> pretty
> lightweight solution, and easier than the old standby of compiling  
> your
> own Python to get isolation from the global one, whichstill highly
> recommended:  I build my own Python, and then use a separate  
> virtualenv
> for each project.
>

The idea behind Gary's branch (http://svn.zope.org/zc.buildout/branches/gary-4-include-site-packages 
) is that unlike the --no-site-packages option of virtualenv, which is  
all-or-nothing proposition, you would be able to include site-package  
locations in Buildout's script generation, but care would be taken  
that if distributions are selected from a site-package location to  
make sure that when site-package locations are included on sys.path,  
those locations don't overshadow any other paths pointing explicitly  
to already picked versions of distributions. e.g. If I was using  
Apple's System Python on Leopard (10.5), then site-packages includes  
zope.interface 3.3.0 and bdist_mpkg 0.4.3. If I wanted to pick  
'zope.interface == 3.3.0' and 'bdist_mpkg == 0.4.4', then currently  
Buildout could generate a path modification that looks like:

sys.path[0:0] = [
   '/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/ 
lib/python',
   '/Users/kteague/buildouts/shared/eggs/bdist_mpkg-0.4.4-py2.5.egg',
]

Where that System path contains bdist_mpkg 0.4.3. The ordering of  
whether the site-package location is put before or after version- 
specific paths is currently dependant upon the ordering in the  
install_requires field (so you get the correct versions importable if  
those distributions which are picked from site-packages are listed  
after the non-site-package picked versions!) - obviously this is just  
a side-effect of the current path manipulation implementation.

One would assume that making this change is fairly easy. Just do a  
diff between normal sys.path and the site-package free sys.path when  
Python is launched with the -S flag. Which Gary's code does, but the  
script generation in Gary's branch right now also accounts for the  
fact that *.pth files have been processed, and that you are allowed to  
have import statements executed when *.pth files are processed, so he  
is generating scripts which also clean sys.modules, and then re-add  
site-packages locations with site.addsitedir(location) so that .pth  
files are properly re-processed. Which is pretty fancy, and probably  
"Does the Right Thing (TM)", but also greatly clutters up the  
generated scripts.

I quite like having script generation generate scripts which are still  
reasonably compact (I often open generated scripts to see what  
Buildout is doing, or sometimes edit them to hand-pick a different egg  
if I want to quickly try out a different working set) and I also  
wonder how much overhead this additional processing adds (I guess this  
depends upon how much you have in site-packages). So perhaps if there  
was some option to still generate scripts using the existing style of  
script generation - maybe a "i-keep-my-site-packages-clean=true"  
option ... i dunno, perhaps the other way 'work-around-site-package- 
madness-in-script-generation=true'  ... or just merge Buildout and  
VirtualEnv into one monolithic project so that you don't need to  
install two tools just to be able to use Buildout with a dirty  
Python!  (rawr!)

Anyways, for those distributions which are tough to install, I think  
some people will find this branch quite handy in that they can apt-get  
the tough to install distributions, and then safely include those  
distributions in working sets composed by Buildout.