[Distutils] Setting environment variables before building eggs w/ extensions using zc.buildout

[Jim Fulton]

On Mar 7, 2007, at 2:07 PM, Nathan R. Yergler wrote:

I meant to reply earlier. Sorry.

> I had a problem come up today that I've run into before, and I'd like
> to figure out a way to prevent it from sucking my time down in the
> future.  Several apps we run at Creative Commons use lxml
> (http://codespeak.net/lxml) for XML, XSLT and XPath processing.  lxml
> builds on libxml2, and provides an element tree interface, plus
> additional capabilities.
>
> We use zc.buildout to assemble our server-side applications, and that
> has generally been a huge sanity saver.  However, lxml can rob some of
> that sanity periodically.  If a system like one of our CentOS machines
> has an older version of libxml2 installed in /usr/lib, the extension
> building process will link against it without complaint.  That is,
> until you try to do something added in the later, recommended version
> of libxml2 like XSLT.  Once you scratch your head and remember what
> the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib,
> and then re-run the buildout.
>
> Does zc.buildout currently have any way to set an environment variable
> during it's run?

No.

>   I didn't see anything at
> http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if
> there was something at the zc.buildout level (as opposed to the recipe
> level) that would do the trick.  The quick/dirty thing I'd do is
> export the appropriate value of LD_LIBRARY_PATH.

I'm a little surprised that this would help at build time.  I'm  
surprised that distutils looks at LD_LIBRARY_PATH when deciding where  
to link from.

Have you looked at:

   http://www.python.org/pypi/zc.recipe.egg#creating-eggs-with- 
extensions-neededing-custom-build-settings

>   The perhaps more
> intelligent thing I'd do is use the cmmi recipe to build a version of
> libxml2 that I *know* has the features I need as part of the buildout,
> and then point LD_LIBRARY_PATH at that.

Maybe. That's what I would do. :)

There are really 2 issues:

- Whenther to rely on your system libraries,

- How to tell distutils where the libraries are, both at build and at  
run time.

The former depends on the library.  I've had problems with this  
particular library on Red Hat-based systems, si we typically build  
our own as part of a buildout.

For the later, the zc.recipe.egg:custom recipe is what we use.  It is  
simplest to use the rpath option at build time.

The rpath option doesn't work for us in production deployments  
because our deployment-install locations are different than our build  
locations, so we use LD_LIBRARY_PATH at run time.  Because we use  
zdaemon and zdaemon lets us specify run-time environment variables,  
this isn't a problem. (Note that a Python script can't set it's own  
LD_LIBRARY_PATH, because it is too late -- the library loader has  
already looked for this before a Python script can set it.  This  
works with zdaemon because zdaemon creates a separate subprocess for  
the application and the environment variables are set before the  
subprocess runs.)

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org