[Catalog-sig] [zc.buildout] Dealing with building (large) libraries

[Jim Fulton]

On Jan 25, 2009, at 8:19 AM, Andreas Jung wrote:
> it had become a common pattern with buildout compiling almost all and
> everything within one buildout. E.g. the Deliverance integration using
> plone.recipe.deliverance downloads and compiles libxml2/libxslt which
> takes a lot of time. In addition we have seen unmotivated
> uninstall/install orgies of parts (possibly the related recipes are to
> blame) causing a lot of turnaround time for developers (and  
> frustration
> about using buildout).

Buildout takes a conservative approach when deciding whether a part  
needs to be reinstalled.  In particular, a change to a part's recipe  
(like a new recipe egg) or a package the recipe depends on (e.g.  
buildout itself) will cause a part to be reinstalled.


> Anyone having similar experiences and/or hints how deal with such  
> larger
> buildouts?

I don't think it's really a question of the size of the buildout so  
much as the expense of individual parts. Many or most parts aren't  
expensive to reinstall.  Certain parts, like those that build a big  
external library can be especially painful.

> We are having a company internal sprint next week where we
> are thinking about a 2-stage buildout for some of our projects where  
> the
> fat parts will be moved to a dedicated buildout configuration and
> installed/maintained as as global resources. This will at least reduce
> the number of pointless uninstall/install cycles.


That's a reasonable approach.  Another approach might be to add an  
option to make buildout less conservative about certain parts.   For  
example, there might be an option to, for a given list of parts to  
only reinstall a part if
an option changes, ignoring changes to the version of the part recipe  
or it's dependencies. Alternatively, we could change buildout to use a  
provided value __buildout_signature__, rather than computing one  
itself if the option is provided. Then, for expensive parts, like one  
building a library, once could simply provide this option, giving a  
static value. I think this would be more effective that managing  
separate buildouts to compute expensive parts.

Jim

--
Jim Fulton
Zope Corporation