[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
tjreedy at udel.edu
Wed Jun 16 21:27:09 CEST 2010
On 6/16/2010 8:20 AM, M.-A. Lemburg wrote:
> Antoine Pitrou wrote:
>> Tarek Ziadé<ziade.tarek<at> gmail.com> writes:
>>> And we happen to have this network already: lots of people
>>> will host a PyPI mirror as soon as it's easy to set one imho.
>> You must be careful that the mirrors are properly managed and administered,
>> though. Having stale/dysfunctioning mirrors is worse than having no mirrors at
>> It is likely that some people will setup a mirror and then "forget" to take care
>> about it. Like our buildbots really.
> Right, it's that administration overhead I was referring to.
> Perhaps we should just let the users decide:
> a) they use the default PyPI access (which we then enhance by
> caching the content in the cloud)
> b) they setup their easy_install or zc.buildout to pull data
> from a mirror network by enabling a configuration option
> Since implementing option b) will require updating existing
> package tools on the client side anyway, the extra configuration
> shouldn't be a problem.
> Option a) requires no changes whatsoever on the client side.
It seems to me that:
If the problem of availability with pypi is anything like the problems
with bugs... and extending 'pypi...' with cloud service could be done
relatively quickly (within a month), then that seems reasonable.
If 'free to psf' mirrors are feasible and needed, then they will still
be useful, especially is high-download regions. Since Amazon's cloud
service is metered on a region by region basis, any off-loading of
demand to regional mirrors will reduce PSF charges. Based on what I have
read in the thread, I would not be surprised if full mirror deployment
takes a year. After that, the cloud service could remain to pick up
slack in a region should the mirror in a region go down.
Any move to incremental update from time-based replacement will benefit
Terry Jan Reedy
More information about the Catalog-SIG