On Aug 27, 2009, at 3:46 AM, Chris Withers wrote:

Gary Poster wrote:

...

While it has the functionality that I believe you want (that is, you can entirely exclude the normal Python site-packages),

Cool :-)

...

we're using it successfully on Launchpad right now to do something similar to what you describe, but different. We allow site- packages through, so we get system lxml and pyopenssl, for instance, but we explicitly get distributions via buildout for the bits that are more closely tied to the application, even if conflicting eggs are installed for the system. We've only had the changes in place for a couple of weeks, but things seem to be *much* smoother with them now.

Indeed. Sounds great. However, what do you do if the system installed libraries are the wrong very (eg: the buildout specifies a particular version, and the one in site-packages isn't it)?

Using a whitelist in your .cfg, some or all of the eggs in site- packages are simply ignored when buildout assembles dependencies (even if they are or claim to be the right versions); and then the eggs that buildout chooses have precedence over site-packages (they come before site-packages in sys.path). Gary

On Aug 31, 2009, at 8:29 AM, Chris Withers wrote:

Gary Poster wrote:

...

...

Indeed. Sounds great. However, what do you do if the system installed libraries are the wrong very (eg: the buildout specifies a particular version, and the one in site-packages isn't it)? Using a whitelist in your .cfg,

What does this mean/do/look like?

First, to repeat: Jim has not yet had a chance to review (and now he's waiting for me to do a merge with trunk, and I'm also using that to test the branch out on the upcoming Ubuntu release because I got a bug report), and this may change. That said... In the simplest case: allowed-eggs-from-site-packages = That means you don't allow any eggs from site packages. So, this means that if one of your direct or indirect dependencies is docutils, and docutils is installed as an egg in your system, when the eggs for your buildout are assembled, the system egg will be ignored, and buildout will go on the hunt for one someplace else. The typical result is that the egg that it finds will be earlier in the site path, so it has precedence in scripts and so on. Meanwhile, dependencies that are not part of your dependencies (like, say, pyopenssl) are still available from your standard site-packages. Alternatively, consider this. allowed-eggs-from-site-packages = lxml That is, you only allow the lxml egg from site packages. All the rest will be ignored, as per the description above. Here are the docs: allowed-eggs-from-site-packages Sometimes you need or want to control what eggs from site- packages are used. The allowed-eggs-from-site-packages option allows you to specify a whitelist of project names that may be included from site- packages. You can use globs to specify the value. It defaults to a single value of '*', indicating that any package may come from site-packages. Here's a usage example:: [buildout] ... allowed-eggs-from-site-packages = demo bigdemo zope.* This option interacts with the ``include-site-packages`` option in the following ways. If ``include-site-packages`` is true, then ``allowed-eggs-from-site-packages`` filters what eggs from site- packages may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an empty list, then no eggs from site-packages are chosen, but site- packages will still be included at the end of path lists. If ``include-site-packages`` is false, the value of ``allowed-eggs-from-site-packages`` is irrelevant. The same pattern holds true for the ``include-site-packages-for- buildout`` option, except only the bin/buildout script is affected by that interaction.

...

some or all of the eggs in site-packages are simply ignored when buildout assembles dependencies (even if they are or claim to be the right versions); and then the eggs that buildout chooses have precedence over site-packages (they come before site-packages in sys.path).

Okay, but what if I don't know (or more accurately: don't want to have to care ;-) ) what's in site-packages? What happens if there's no whitelist and either buildout can't figure out what version of the package is installed or it's the wrong version?

Simplest answer: if you are not using any new features, this usually fails, as before. I'm loathe to significantly change the default behavior in 1.x release, and someone may be doing virtualenv/buildout dances that rely on the current behavior with site-packages. The next simplest answer is to use the new "include-site-packages = false" option. That's a tiny bit like using a virtualenv for your buildout, in that you are using the system executable but a clean site- packages. The last answer is to leave "include-site-packages = true" (the default), and use the "allowed-eggs-from-site-packages" whitelist I describe above. We currently use an empty whitelist, so we get bits like pyopenssl from the system, but buildout manages all of the application eggs. It's a compromise that is working well for us so far. Gary

On Thu, 2009-08-27 at 08:36 -0400, Jim Fulton wrote:

...

3) exist any plan for buildout to obtain the same isolation level than virtualenv ? when ?

As Gary mentioned, yes. I hope this will be in 1.5, within a few weeks (maybe days :).

this is, that the buildout will ignore the system python path ? this is the final purpose that i want to obtain right now.

Sure. The simplest, IMO, is to use a clean Python built from source that you keep clean and share among your various projects. We deploy to CentOS-based systems and deploy "cleanpython" RPMs (for both 2.4 and 2.6) along side the Red Hat supplied installs. We develop on Mac and Ubuntu and typically have clean Python builds that we share among our various projects.

we deploy in centos servers, with python 2.5 build from source in /usr/local (and i am trying jython 2.5 for some apps) but we prefer virtualenv for isolate apps.

Of course, you can create a virtualenv and keep it clean and use it for multiple buildouts. This is the rough equivalent of of creating a clean Python. Buildout never installs into site-packages, so if you create a clean virtualenv, buildout won't dirty it.

well, i think that i am going to write the recipe for: 1) build a virtualenv in the same buildout directory (with several options, python interpreter version, --no-site-packages, etc) 2) activate the virtualenv 3) restart the buildout running with the python interpreter from the buildout i think that can be a good exercise for understand how to write recipes for buildout. i found docs for recipe but very few for extensions, can u give me some indications where i can search for this ? thx in advance and i wait for 1.5 ^_^ -- Yonsy Solis Aureal Systems (mov) 989-124-141 (www) http://www.aureal.com.pe

On Fri, 2009-08-28 at 23:36 -0400, Benji York wrote:

On Fri, Aug 28, 2009 at 11:15 PM, Yonsy Solis wrote:

...

i found docs for recipe but very few for extensions, can u give me some indications where i can search for this ?

http://pypi.python.org/pypi?:action=browse&show=all&c=512

err. i refer to documentation to develop extensions, i found documentation for develop recipes, btw recipes == extensions in buildout ? -- Yonsy Solis Aureal Systems (mov) 989-124-141 (www) http://www.aureal.com.pe

Benji York wrote:

On Fri, Aug 28, 2009 at 11:39 PM, Yonsy Solis wrote:

...

btw recipes == extensions in buildout ?

Yes.

No. Buildout allows extensions as well as recipes. Recipes are just used by buildout. Extensions (such as lovely.buildouthttp) change how buildout works. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

we deploy in centos servers, with python 2.5 build from source in /usr/local (and i am trying jython 2.5 for some apps) but we prefer virtualenv for isolate apps. Hi Younsy, I'd love to hear about how you do with Jython and your apps with virtualenv, distutils, etc - especially if you run into any

On Fri, Aug 28, 2009 at 11:15 PM, Yonsy Solis wrote: trouble. -Frank

You might give a try to a project of mine: http://pyvm.sourceforge.net It provides what people expect from a clean build of python. Regards, Antonio

On Wed, Aug 26, 2009 at 8:38 PM, Yonsy Solis wrote: ...

...

3) exist any plan for buildout to obtain the same isolation level than virtualenv ? when ?

As Gary mentioned, yes. I hope this will be in 1.5, within a few weeks (maybe days :).

...

4) do you see any other alternative to obtain the same effect (we need to test apps between python 2.5 and python 2.6, we see than the buildout dont ignore the global site-package, some dev machines are really messed in their python interpreters used and the eggs/modules/installed in their systems, and we love to deploy the apps inside venvs with mod_wsgi, no more hell with pythonpath)

Sure. The simplest, IMO, is to use a clean Python built from source that you keep clean and share among your various projects. We deploy to CentOS-based systems and deploy "cleanpython" RPMs (for both 2.4 and 2.6) along side the Red Hat supplied installs. We develop on Mac and Ubuntu and typically have clean Python builds that we share among our various projects.

Of course, you can create a virtualenv and keep it clean and use it for multiple buildouts. This is the rough equivalent of of creating a clean Python. Buildout never installs into site-packages, so if you create a clean virtualenv, buildout won't dirty it.

Jim