[Distutils] virtualenv and buildout integration ?

[Gary Poster]

On Aug 31, 2009, at 8:29 AM, Chris Withers wrote:

> Gary Poster wrote:
>>> Indeed. Sounds great. However, what do you do if the system  
>>> installed libraries are the wrong very (eg: the buildout specifies  
>>> a particular version, and the one in site-packages isn't it)?
>> Using a whitelist in your .cfg,
>
> What does this mean/do/look like?

First, to repeat: Jim has not yet had a chance to review (and now he's  
waiting for me to do a merge with trunk, and I'm also using that to  
test the branch out on the upcoming Ubuntu release because I got a bug  
report), and this may change.  That said...

In the simplest case:

allowed-eggs-from-site-packages =

That means you don't allow any eggs from site packages.  So, this  
means that if one of your direct or indirect dependencies is docutils,  
and docutils is installed as an egg in your system, when the eggs for  
your buildout are assembled, the system egg will be ignored, and  
buildout will go on the hunt for one someplace else.  The typical  
result is that the egg that it finds will be earlier in the site path,  
so it has precedence in scripts and so on.

Meanwhile, dependencies that are not part of your dependencies (like,  
say, pyopenssl) are still available from your standard site-packages.

Alternatively, consider this.
allowed-eggs-from-site-packages =
     lxml

That is, you only allow the lxml egg from site packages.  All the rest  
will be ignored, as per the description above.

Here are the docs:

allowed-eggs-from-site-packages
     Sometimes you need or want to control what eggs from site- 
packages are
     used. The allowed-eggs-from-site-packages option allows you to  
specify a
     whitelist of project names that may be included from site- 
packages.  You
     can use globs to specify the value.  It defaults to a single  
value of '*',
     indicating that any package may come from site-packages.

     Here's a usage example::

         [buildout]
         ...

         allowed-eggs-from-site-packages =
             demo
             bigdemo
             zope.*

     This option interacts with the ``include-site-packages`` option  
in the
     following ways.

     If ``include-site-packages`` is true, then
     ``allowed-eggs-from-site-packages`` filters what eggs from site- 
packages
     may be chosen.  Therefore, if ``allowed-eggs-from-site-packages``  
is an
     empty list, then no eggs from site-packages are chosen, but site- 
packages
     will still be included at the end of path lists.

     If ``include-site-packages`` is false, the value of
     ``allowed-eggs-from-site-packages`` is irrelevant.

     The same pattern holds true for the ``include-site-packages-for- 
buildout``
     option, except only the bin/buildout script is affected by that
     interaction.


>
>> some or all of the eggs in site-packages are simply ignored when  
>> buildout assembles dependencies (even if they are or claim to be  
>> the right versions); and then the eggs that buildout chooses have  
>> precedence over site-packages (they come before site-packages in  
>> sys.path).
>
> Okay, but what if I don't know (or more accurately: don't want to  
> have to care ;-) ) what's in site-packages? What happens if there's  
> no whitelist and either buildout can't figure out what version of  
> the package is installed or it's the wrong version?

Simplest answer: if you are not using any new features, this usually  
fails, as before.  I'm loathe to significantly change the default  
behavior in 1.x release, and someone may be doing virtualenv/buildout  
dances that rely on the current behavior with site-packages.

The next simplest answer is to use the new "include-site-packages =  
false" option.  That's a tiny bit like using a virtualenv for your  
buildout, in that you are using the system executable but a clean site- 
packages.

The last answer is to leave "include-site-packages = true" (the  
default), and use the "allowed-eggs-from-site-packages" whitelist I  
describe above.  We currently use an empty whitelist, so we get bits  
like pyopenssl from the system, but buildout manages all of the  
application eggs.  It's a compromise that is working well for us so far.

Gary