[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

Terry Reedy tjreedy at udel.edu
Wed Jun 16 21:27:09 CEST 2010

On 6/16/2010 8:20 AM, M.-A. Lemburg wrote:
> Antoine Pitrou wrote:
>> Tarek Ziadé<ziade.tarek<at>  gmail.com>  writes:
>>> And we happen to have this network already: lots of people
>>> will host a PyPI mirror as soon as it's easy to set one imho.
>> You must be careful that the mirrors are properly managed and administered,
>> though. Having stale/dysfunctioning mirrors is worse than having no mirrors at
>> all.
>> It is likely that some people will setup a mirror and then "forget" to take care
>> about it. Like our buildbots really.
> Right, it's that administration overhead I was referring to.
> Perhaps we should just let the users decide:
> a) they use the default PyPI access (which we then enhance by
>     caching the content in the cloud)
> b) they setup their easy_install or zc.buildout to pull data
>     from a mirror network by enabling a configuration option
> Since implementing option b) will require updating existing
> package tools on the client side anyway, the extra configuration
> shouldn't be a problem.
> Option a) requires no changes whatsoever on the client side.

It seems to me that:

If the problem of availability with pypi is anything like the problems 
with bugs... and extending 'pypi...' with cloud service could be done 
relatively quickly (within a month), then that seems reasonable.

If 'free to psf' mirrors are feasible and needed, then they will still 
be useful, especially is high-download regions. Since Amazon's cloud 
service is metered on a region by region basis, any off-loading of 
demand to regional mirrors will reduce PSF charges. Based on what I have 
read in the thread, I would not be surprised if full mirror deployment 
takes a year. After that, the cloud service could remain to pick up 
slack in a region should the mirror in a region go down.

Any move to incremental update from time-based replacement will benefit 
either system.

Terry Jan Reedy

More information about the Catalog-SIG mailing list