[Distutils] buildout download cache

[Rafael Monnerat]

Hi,

I made a patch to zc.buildout.download which introduces a very simple 
"network cache" or "cache into network".  So, I tried to keep this as 
simple and generic as buildout is. Basically, with the patch the 
download of a file, follow this order:

1. Try Local Cache (no change from original)
2. Try network cache (I explain bellow)
3. Try original URL
4. Post file data to network cache (pure HTTP)

So, the network cache is just one URL where files are placed (can be any 
simple HTTP) and identified by file MD5 like this:

  GET http://my.company.shared.cache/md5_provided_for_the_file

The cache update is done by a simple post to same adress:

  POST http://my.company.shared.cache/md5_provided_for_the_file < data

As I'm familiar with ERP5, I implemented a very simple way to handle 
this cache, but I can also contribute with a simpler solution like 
eggproxy do for eggs. If you think my patch is useful and ok.

As I don't have access to svn to make a branch, I'm attaching the patch.

With network cache, I think people can share downloads into private 
networks or prevent your build is break when some source is unavailable.

If you consider this behaviour inapropriate for the core of buildout, 
but appropriated to be an buildout extension, let me know.

Regards,
Rafael Monnerat

On 11-01-2011 06:03, Thomas Lotze wrote:
> rafael wrote:
>
>> We are planning to extend buildout download API to provide a distributed
>> automatic
>> packaging and caching system so that even if original source web site is
>> down, the buildout process can stil run. This could also be useful to
>> build software in
>> secured networks.
> I'd strongly suggest keeping this logic out of the download API. It sounds
> like something that may potentially grow a lot more complex than a simple
> "download this URL, with or without using a cache" gesture.
>
> In my opinion, a distributed packaging system is application logic from
> the perspective of a generic framework such as zc.buildout. It might be
> implemented by a recipe, some library on top of the download API or some
> other mechanism altogether, but it should neither complicate the semantics
> of the existing download API nor add a new one to the zc.buildout code
> base.
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: zc.buildout.download.patch
Type: text/x-diff
Size: 3063 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20110114/f1121c86/attachment.patch>