[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

[Terry Reedy]

On 6/16/2010 8:20 AM, M.-A. Lemburg wrote:
> Antoine Pitrou wrote:
>> Tarek Ziadé<ziade.tarek<at>  gmail.com>  writes:
>>>
>>> And we happen to have this network already: lots of people
>>> will host a PyPI mirror as soon as it's easy to set one imho.
>>
>> You must be careful that the mirrors are properly managed and administered,
>> though. Having stale/dysfunctioning mirrors is worse than having no mirrors at
>> all.
>> It is likely that some people will setup a mirror and then "forget" to take care
>> about it. Like our buildbots really.
>
> Right, it's that administration overhead I was referring to.
>
> Perhaps we should just let the users decide:
>
> a) they use the default PyPI access (which we then enhance by
>     caching the content in the cloud)
>
> b) they setup their easy_install or zc.buildout to pull data
>     from a mirror network by enabling a configuration option
>
> Since implementing option b) will require updating existing
> package tools on the client side anyway, the extra configuration
> shouldn't be a problem.
>
> Option a) requires no changes whatsoever on the client side.

It seems to me that:

If the problem of availability with pypi is anything like the problems 
with bugs... and extending 'pypi...' with cloud service could be done 
relatively quickly (within a month), then that seems reasonable.

If 'free to psf' mirrors are feasible and needed, then they will still 
be useful, especially is high-download regions. Since Amazon's cloud 
service is metered on a region by region basis, any off-loading of 
demand to regional mirrors will reduce PSF charges. Based on what I have 
read in the thread, I would not be surprised if full mirror deployment 
takes a year. After that, the cloud service could remain to pick up 
slack in a region should the mirror in a region go down.

Any move to incremental update from time-based replacement will benefit 
either system.

Terry Jan Reedy