[Distutils] zc.buildout

[Jim Fulton]

On Aug 8, 2007, at 1:41 PM, Miles wrote:

Sorry to take so long to reply....

Your earlier post today reminded me that I needed to reply.  I hope  
that this will answer both your original post and today's.

> Hi,
>
> We have some a zc.buildout-based deployment system, and we're  
> looking at
> how we can make the process slightly easier for moving between
> development, testing and production environments.
>
> I was wondering if others have a pattern for setting up a buildout in
> this way?  I am imagining that this is better done by running the
> different steps, but putting the outputs in different locations (e.g.
> for testing, put apache configurations in some directory in the test
> tree), rather than trying to selectively run steps (e.g. for  
> testing, i
> do not want apache configurations set up).  Is there a useful
> buildout.cfg pattern that covers this?
>
> Also, one other use-case that we've got is to be able to group parts
> together into sets.  So, for example, I'd lie to be able to do  
> something
> like:
>
> [buildout]
> parts = awstats
>
> [awstats]
> parts = awstats-download apacheconf-install crontab-install
>
> which would tell buildout to run the three parts listed, rather than
> remembering all 3 specific parts in the parts list.  The advantage  
> being
> to ensure the bits are always run together.
>
> Has anyone done anything like this at all?

Yes, In practice, we find we need 2 modes, development, and  
deployment. (Staging is just a special case of deployment for us and  
I expect, many others.)

We use buildout.cfg for development, which includes most testing.  We  
deploy using RPMs.  We have an rpm.cfg and, because we create  
relocatable RPMs, an rpmpost.cfg that fixes up paths.  We also have a  
separate configuration file that configures software instances,  
rather than building software.

The buildout.cfg, and rpm.cfg, are run by the developer.  The later  
is run via an RPM spec file.

The rpmpost.cfg is run automatically by root at part of the software  
install.  This uses buildout on offline mode so that no new software  
can be installed via the network.  Basically, it's in the RPM (or  
already on the system) or it's not available.  This is for security  
purposes, since the buildout is run as root.

After the software is installed, we run the buildout again using yet  
another configuration file.  This configuration file sets up  
"instances" of the software such as database and application  
processes.  This is run as root and in offline mode.  It writes to  
places like /etc, /var/log, and so on.

As Martin suggested, you can use the fact that buildout  
configurations can extend other buildout configurations to factor  
commonalities. The various configuration files often extend a common  
one, but they don't need to.

I've been meaning to write this up in detail with an example, but  
haven't been able to find time to. :(

I'll at least put an example together without words so that people  
can maybe figure out what we've done until I can find the time to  
write words to go with it.

On Aug 20, 2007, at 4:42 PM, Miles wrote:
...
> Thinking about it a bit more, what originally sparked off the post  
> was a
> need to run some steps as root (e.g. to install startup scripts in
> /etc/init.d), and others as a different user (zope files owned by the
> zope user).
>
> At the moment, we tackle this by (for example):
>
> $ su
> $ bin/buildout -c buildout-root.cfg
> $ exit
> $ su zope
> $ bin/buildout -c buidlout-zope.cfg
> ...
>
> However, this starts to get complicated as more users are involved!

Why do you need more users to be involved.  In fact, I don't see a  
need to have more than 2 users involved, the developer and root.

> Plus, each step cannot easily use the configuration from a previous  
> one,
> so there's some repetition.

This is mostly a matter of factoring.  In fact, it often is possible  
to reuse configuration from previous steps.  (There are of course,  
cases where this can be more convenient.)

> Do you have a pattern that covers this at all?

See if my description of what we do above makes any sense.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org