Setting environment variables before building eggs w/ extensions using zc.buildout
I had a problem come up today that I've run into before, and I'd like to figure out a way to prevent it from sucking my time down in the future. Several apps we run at Creative Commons use lxml (http://codespeak.net/lxml) for XML, XSLT and XPath processing. lxml builds on libxml2, and provides an element tree interface, plus additional capabilities. We use zc.buildout to assemble our server-side applications, and that has generally been a huge sanity saver. However, lxml can rob some of that sanity periodically. If a system like one of our CentOS machines has an older version of libxml2 installed in /usr/lib, the extension building process will link against it without complaint. That is, until you try to do something added in the later, recommended version of libxml2 like XSLT. Once you scratch your head and remember what the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib, and then re-run the buildout. Does zc.buildout currently have any way to set an environment variable during it's run? I didn't see anything at http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if there was something at the zc.buildout level (as opposed to the recipe level) that would do the trick. The quick/dirty thing I'd do is export the appropriate value of LD_LIBRARY_PATH. The perhaps more intelligent thing I'd do is use the cmmi recipe to build a version of libxml2 that I *know* has the features I need as part of the buildout, and then point LD_LIBRARY_PATH at that. Or has anyone else run into a similar situation, who might have some insight? Thanks, Nathan
Nathan R. Yergler wrote:
I had a problem come up today that I've run into before, and I'd like to figure out a way to prevent it from sucking my time down in the future. Several apps we run at Creative Commons use lxml (http://codespeak.net/lxml) for XML, XSLT and XPath processing. lxml builds on libxml2, and provides an element tree interface, plus additional capabilities.
We use zc.buildout to assemble our server-side applications, and that has generally been a huge sanity saver. However, lxml can rob some of that sanity periodically. If a system like one of our CentOS machines has an older version of libxml2 installed in /usr/lib, the extension building process will link against it without complaint. That is, until you try to do something added in the later, recommended version of libxml2 like XSLT. Once you scratch your head and remember what the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib, and then re-run the buildout.
Does zc.buildout currently have any way to set an environment variable during it's run? I didn't see anything at http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if there was something at the zc.buildout level (as opposed to the recipe level) that would do the trick. The quick/dirty thing I'd do is export the appropriate value of LD_LIBRARY_PATH. The perhaps more intelligent thing I'd do is use the cmmi recipe to build a version of libxml2 that I *know* has the features I need as part of the buildout, and then point LD_LIBRARY_PATH at that.
Or has anyone else run into a similar situation, who might have some insight?
I would be interested in something like this, specifically for building Deliverance. Deliverance does have a buildout, that attempts to do this, but the deliverance buildout is pretty rough. Or rather, it's not easily re-usable, so if you want to build deliverance as part of a bigger project, it can be tricky. I'd be interested in seeing and/or helping out with such a solution because I need to build deliverance as part of this project anyway, and I'd like an elegant way to do it. See http://www.openplans.org/projects/deliverance and http://codespeak.net/svn/z3/deliverance/buildout/trunk/. Martin
Martin Aspeli wrote:
Nathan R. Yergler wrote:
I had a problem come up today that I've run into before, and I'd like to figure out a way to prevent it from sucking my time down in the future. Several apps we run at Creative Commons use lxml (http://codespeak.net/lxml) for XML, XSLT and XPath processing. lxml builds on libxml2, and provides an element tree interface, plus additional capabilities.
We use zc.buildout to assemble our server-side applications, and that has generally been a huge sanity saver. However, lxml can rob some of that sanity periodically. If a system like one of our CentOS machines has an older version of libxml2 installed in /usr/lib, the extension building process will link against it without complaint. That is, until you try to do something added in the later, recommended version of libxml2 like XSLT. Once you scratch your head and remember what the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib, and then re-run the buildout.
Does zc.buildout currently have any way to set an environment variable during it's run? I didn't see anything at http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if there was something at the zc.buildout level (as opposed to the recipe level) that would do the trick. The quick/dirty thing I'd do is export the appropriate value of LD_LIBRARY_PATH. The perhaps more intelligent thing I'd do is use the cmmi recipe to build a version of libxml2 that I *know* has the features I need as part of the buildout, and then point LD_LIBRARY_PATH at that.
Or has anyone else run into a similar situation, who might have some insight?
I would be interested in something like this, specifically for building Deliverance.
Deliverance does have a buildout, that attempts to do this, but the deliverance buildout is pretty rough. Or rather, it's not easily re-usable, so if you want to build deliverance as part of a bigger project, it can be tricky.
I'd be interested in seeing and/or helping out with such a solution because I need to build deliverance as part of this project anyway, and I'd like an elegant way to do it.
See http://www.openplans.org/projects/deliverance and http://codespeak.net/svn/z3/deliverance/buildout/trunk/.
Martin
fyi, the deliverance.buildout will soon go away -w -- ------ d. whit morriss ------ - senior engineer, opencore - - http://www.openplans.org - - m: 415-710-8975 - "If you don't know where you are, you don't know anything at all" Dr. Edgar Spencer, Ph.D., 1995
Martin Aspeli wrote:
whit wrote:
fyi, the deliverance.buildout will soon go away
Well, I don't think it has much value for re-usability of Deliverance anyway.
Do you have an alternate distribution mechanism? Perhaps one we could leverage inside a buildout? :)
Martin
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
indeed... it will be replaced by a buildit driven task that could be also driven from buildout. not sure on the eta. -w -- ------ d. whit morriss ------ - senior engineer, opencore - - http://www.openplans.org - - m: 415-710-8975 - "If you don't know where you are, you don't know anything at all" Dr. Edgar Spencer, Ph.D., 1995
On Mar 7, 2007, at 2:07 PM, Nathan R. Yergler wrote: I meant to reply earlier. Sorry.
I had a problem come up today that I've run into before, and I'd like to figure out a way to prevent it from sucking my time down in the future. Several apps we run at Creative Commons use lxml (http://codespeak.net/lxml) for XML, XSLT and XPath processing. lxml builds on libxml2, and provides an element tree interface, plus additional capabilities.
We use zc.buildout to assemble our server-side applications, and that has generally been a huge sanity saver. However, lxml can rob some of that sanity periodically. If a system like one of our CentOS machines has an older version of libxml2 installed in /usr/lib, the extension building process will link against it without complaint. That is, until you try to do something added in the later, recommended version of libxml2 like XSLT. Once you scratch your head and remember what the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib, and then re-run the buildout.
Does zc.buildout currently have any way to set an environment variable during it's run?
No.
I didn't see anything at http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if there was something at the zc.buildout level (as opposed to the recipe level) that would do the trick. The quick/dirty thing I'd do is export the appropriate value of LD_LIBRARY_PATH.
I'm a little surprised that this would help at build time. I'm surprised that distutils looks at LD_LIBRARY_PATH when deciding where to link from. Have you looked at: http://www.python.org/pypi/zc.recipe.egg#creating-eggs-with- extensions-neededing-custom-build-settings
The perhaps more intelligent thing I'd do is use the cmmi recipe to build a version of libxml2 that I *know* has the features I need as part of the buildout, and then point LD_LIBRARY_PATH at that.
Maybe. That's what I would do. :) There are really 2 issues: - Whenther to rely on your system libraries, - How to tell distutils where the libraries are, both at build and at run time. The former depends on the library. I've had problems with this particular library on Red Hat-based systems, si we typically build our own as part of a buildout. For the later, the zc.recipe.egg:custom recipe is what we use. It is simplest to use the rpath option at build time. The rpath option doesn't work for us in production deployments because our deployment-install locations are different than our build locations, so we use LD_LIBRARY_PATH at run time. Because we use zdaemon and zdaemon lets us specify run-time environment variables, this isn't a problem. (Note that a Python script can't set it's own LD_LIBRARY_PATH, because it is too late -- the library loader has already looked for this before a Python script can set it. This works with zdaemon because zdaemon creates a separate subprocess for the application and the environment variables are set before the subprocess runs.) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Late reply... On 3/8/07, Jim Fulton <jim@zope.com> wrote:
On Mar 7, 2007, at 2:07 PM, Nathan R. Yergler wrote:
I meant to reply earlier. Sorry.
I had a problem come up today that I've run into before, and I'd like to figure out a way to prevent it from sucking my time down in the future. Several apps we run at Creative Commons use lxml (http://codespeak.net/lxml) for XML, XSLT and XPath processing. lxml builds on libxml2, and provides an element tree interface, plus additional capabilities.
We use zc.buildout to assemble our server-side applications, and that has generally been a huge sanity saver. However, lxml can rob some of that sanity periodically. If a system like one of our CentOS machines has an older version of libxml2 installed in /usr/lib, the extension building process will link against it without complaint. That is, until you try to do something added in the later, recommended version of libxml2 like XSLT. Once you scratch your head and remember what the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib, and then re-run the buildout.
Does zc.buildout currently have any way to set an environment variable during it's run?
No.
I didn't see anything at http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if there was something at the zc.buildout level (as opposed to the recipe level) that would do the trick. The quick/dirty thing I'd do is export the appropriate value of LD_LIBRARY_PATH.
I'm a little surprised that this would help at build time. I'm surprised that distutils looks at LD_LIBRARY_PATH when deciding where to link from.
Have you looked at:
http://www.python.org/pypi/zc.recipe.egg#creating-eggs-with- extensions-neededing-custom-build-settings
The perhaps more intelligent thing I'd do is use the cmmi recipe to build a version of libxml2 that I *know* has the features I need as part of the buildout, and then point LD_LIBRARY_PATH at that.
Maybe. That's what I would do. :)
There are really 2 issues:
- Whenther to rely on your system libraries,
- How to tell distutils where the libraries are, both at build and at run time.
The former depends on the library. I've had problems with this particular library on Red Hat-based systems, si we typically build our own as part of a buildout.
For the later, the zc.recipe.egg:custom recipe is what we use. It is simplest to use the rpath option at build time.
The rpath option doesn't work for us in production deployments because our deployment-install locations are different than our build locations, so we use LD_LIBRARY_PATH at run time. Because we use zdaemon and zdaemon lets us specify run-time environment variables, this isn't a problem. (Note that a Python script can't set it's own LD_LIBRARY_PATH, because it is too late -- the library loader has already looked for this before a Python script can set it. This works with zdaemon because zdaemon creates a separate subprocess for the application and the environment variables are set before the subprocess runs.)
Ah; luckily we're using zdaemon as well, so I'll go down that path. I suppose I could combine what Deliverance does with zdaemon and have a buildout which: * builds libxml2 and libxslt * builds lxml against them * updates my zdaemon.conf to include the resulting lib path
Jim
-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
participants (4)
-
Jim Fulton -
Martin Aspeli -
Nathan R. Yergler -
whit