[Distutils] buildout removes parts on any change

Jim Fulton jim at zope.com
Thu Sep 2 22:23:21 CEST 2010

On Thu, Sep 2, 2010 at 12:50 PM, felix <crucialfelix at gmail.com> wrote:
> does this fairly represent the policy :  buildout removes all files
> installed by parts anytime the top parts/develop/egg section changes because
> "paranoid is safer".

Basically, a part is reinstalled (uninstalled and then installed) any time
the the recipe or any of its dependencies changes and any time
any of the data read by the recipe changes.  The default policy
doesn't remove all files installed by a part, only those reported by the part.
Some recipes intentionally don't report all of the paths they install.

> I can imagine there would be problems, especially when parts depend on other
> parts. (checking versions etc)

>From the buildout documentation (such as it is):

"When buildout is run, it saves configuration data for installed parts
in a file named ".installed.cfg". In subsequent runs, it compares
part-configuration data stored in the .installed.cfg file and the
part-configuration data loaded from the configuration files as
modified by recipe constructors to decide if the configuration of a
part has changed. If the configuration has changed, or if the recipe
has changed, then the part is uninstalled and reinstalled. The
buildout only looks at the part's options, so any data used to
configure the part needs to be reflected in the part's options. It is
the job of a recipe constructor to make sure that the options include
all relevant data."

You might find it interesting to look at .installed.cfg to get more
insights into exactly what information is used.

> On Thu, Sep 2, 2010 at 5:48 PM, Jim Fulton <jim at zope.com> wrote:
>> Buildout provides a default policy of removing installed files and
>> directories *reported by the install method*.  I think this is a
>> reasonable default policy.  It also allows a part to be uninstalled
>> even if the original recipe code is unavalable for some reason.
> Right, and I've run into that.  but actually at that time I had renamed a
> develop egg and the buildout failed because the original was missing.
> maybe it works for parts but not for develop ?
>> A recipe author can completely replace this default policy by:
>> - Not reporting any installed files in the install method of an
>>  install recipe, and by
>> - providing an uninstall recipe (entry point) that does whatever
>>  uninstall logic seems desireable.
> this would work, but it would be a hack and would be a problem when it
> really
> does need to get uninstalled.

True.  I guess a better approach might be to supply a re-install
hook (either a new entry point or a installer method) that, if
present, would get called *instead* of uninstalling and
installing. I'm not sure that any recipe writers would want to work
that hard to solve this particular problem.


>> - We use buildouts to install/update configuration files independent of
>>  software.
>>  The recipes needed by this are installed as part of the software
>>  RPM,
> nice, I get it now
> does the second buildout get run by the RPM's post-install scripts ?

No. The main idea is that configuration and software configuration are
separate activities.  It always annoys me when I install apache or a
relational database software and, as a result, there are servers
*running* on my system. I don't mind help getting things running, but
I'd much rather that be a separate step. I'd also like to have more
than one shot at getting things running.

> or do you just install it, then run the config-file buildouts ?

Yes.  We also have scripts that make this a bit easier based on our
deployment patterns.  These scripts look in standard places in the
file system for config files, and runs buildout with options that tell
it where to find recipes installed with an application and is prevented
from trying to download software.

> I don't know much about rpm

The idea isn't really rpm (or even system packaging) specific.

> I've had blow ups when something went wrong using just buildout and then
> there isn't a safe way to back out.
> tried to add lxml the other day and it failed to build.
> so this is definitely a solution for that.

Doing the building ahead of time is a plus on multiple levels.
Using systems packaging tools also helps your relationship with your
sysadmins. :)

For the configuration part, it would be nice if buildout had better
tools to help you back out of problems or avoid them. For example, a
dry-run option that told you what would be changed by running buildout
would be nice.


Jim Fulton

More information about the Distutils-SIG mailing list