[Distutils] zc.buildout likes to download eggs...

[Jim Fulton]

Martin Aspeli wrote:
> ... which is great, but say I have 10 buildouts all using all the eggs 
> from zope and plone (either explicitly or as dependencies). I'd rather 
> not replicate this in every buildout.
> 
> I think there are two concerns here:
> 
>   - bandwidth: My ISP gets funny when I go over 20Gb/mo
>   - disk space: A lot of information could be duplicated
> 
> Are there any facilities to share eggs among packages (I have a feeling 
> Jim implied there isn't yet)?

Yes

> If not, what would such a facility look like?

See below.

> For the bandwidth part, perhaps it could be some kind of local 
> repository of eggs (but how to stop zc.buildout being over-zealous and 
> getting eggs again thinking something may be newer?).

By default, buildout tries to get the newest distributions that
meet your requirements.  This is not a bandwidth issue, but given
how slow PyPI is, it is a time issue. (Note that if all of
your requirements have upper bounds and you have distributions
that match the upper bounds, then no network access is performed).

Buildout has an offline mode that prevents checks for newer versions,
but also prevents getting new distributions to meet new requirements.
I've decided to add a "fast" mode that skips the check for newer versions
if existing local distributions satisfy requirements.

> For the disk space part, I'm guessing we'd need a shared "eggs" 
> directory. I imagine I could specify eggs-directory as an absolute or 
> relative path, e.g. ../shared-eggs. I don't know whether one buildout 
> would clear out eggs from another one and so on in this scenario, though.

You can arrange for multiple buildouts to share a common eggs directory.
This is a major win both for space and speed since we can usually
satisfy requirements from a shared eggs directory.

The easiest way to do this is through the user-default configuration file.
Buildout normally reads default configuration options from
.buildout/default.cfg in your home directory (aka ~/.buildout/default.cfg).
Typically people specify the buildout eggs-directory option in this file.
For example, my ~/.buildout/default.cfg file looks like this:

   [buildout]
   eggs-directory = /home/jim/.buildout/eggs

I had to manually create /home/jim/.buildout/eggs when I set this
up. All of my buildouts look for eggs in and download eggs to this
shared directory.

(Note that this directory isn't used for develop eggs or eggs that
are build with custom build options.  It is only used for eggs that
are known to be constant for a given version and thus sharable accross
projects.)

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org