[Distutils] buildout download cache

[rafael at nexedi.com]

Hi,

I give you a better explanation with a Use Case.

We would like to a solution not only for eggs download but for ALL  
kind of downloads that should be done by the buildout. For example  
haproxy setup [1]:

[haproxy]
recipe = hexagonit.recipe.cmmi
url = http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.9.tar.gz
md5sum = 2cbcc95b54c0d803edaa13e7b4aeec25

This recipe uses zc.buildout.download (standard buildout API) and  
requires this that url be online when you run buildout for the first  
time.

Considering that we get files from multiple source, we increase the  
changes that some service be offline, breaking the buildout setup.

The good approach for all downloads should be:

1st: Try Local cache (buildout download API already do this) which are  
not available in the first run.
2nd: Try original URL (buildout download API already do this)
3rd: Try a cache server or "network cache" (what we want to do or know  
if someone already did it)

This will guarantee that buildout could still running even original  
source is offline. It is also preferred that ALL recipes can use this  
network cache and be unified (or available) to download eggs too.

[1]  
https://svn.erp5.org/repos/public/erp5/trunk/buildout/software-profiles/haproxy.cfg

I beleave eggproxy is only usefull for eggs download, right?

Use global download-cache helps to share downloads between your builds  
but it do not help first time users.

Regards,
Rafael Monnerat

Quoting "Marius Gedminas" <marius at pov.lt>:

> On Sun, Jan 09, 2011 at 10:59:18PM +0100, rafael at nexedi.com wrote:
>> We are planning to extend buildout download API to provide a
>> distributed automatic packaging and caching system so that even if
>> original source web site is down, the buildout process can stil run.
>> This could also be useful to build software in secured networks.
>>
>> Do you know of any similar project before we start?
>
> http://pypi.python.org/pypi/collective.eggproxy
>
> (I haven't used it myself yet. When PyPI or original mirrors go down, I
> can always find the necessary package in my development machine's
> ~/.buildout/cache/ and copy it over to where I need it -- because I've
> set a global download-cache in ~/.buildout/default.cfg.)
>
> Marius Gedminas
> --
> Computers are not intelligent.  They only think they are.
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.