[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

[Tarek Ziadé]

On Thu, Jun 17, 2010 at 6:30 AM, Ian Bicking <ianb at colorstudy.com> wrote:
> On Wed, Jun 16, 2010 at 1:37 PM, "Martin v. Löwis" <martin at v.loewis.de>
> wrote:
>>>
>>> It is likely that some people will setup a mirror and then "forget" to
>>> take care
>>> about it. Like our buildbots really.
>>
>>
>> The same can happen to any infrastructure, though. Amazon may decide to
>> change the setup, and then the automated update procedure would break.
>> Of course, they would give advance notice - but then somebody would
>> have to react to that advance notice.
>
> That's not very likely, and if something does change it will be extremely
> well announced and documented.  Amazon is providing a commercial service
> lots of people rely on, their process is formalized and professionalized.
> And if Amazon makes mistakes they'll figure out how to avoid them next time,
> while mirror providers are a rotating crew that is unlikely to easily or
> reliably learn from past mistakes.

if a mirror manager don't do a good job, he'll just be taken out of
the ring after a while.
If we depend 100% on Amazon, and if there's a problem, the mirroring
will be down for the time being and we won't be able to do nothing
about it.

> If we actually understood each time PyPI
> broke and fixed it none of this would be a problem; I'm not blaming anyone
> for that, but it's also not going to change and adding lots of mirror
> systems just adds more systems with exactly the same management problems
> that our current system has.

Yes but the difference is that you don't put all your eggs in the same basket:
it's very unlikely that ALL community mirrors will be down at the same
time, thus
a fall-back mechanism on the client side will raise the availability
automatically.

About Amazon: what will happen in 5 years with their offer ? will our
Cloud-PyPI infrastructure will still work ?  what will be the workload
to maintain it ? You can't
be 100% sure the Python community will be able to dedicate that time.
PyPI works today because it is not forced by a third party to evolve,
it can evolve as its own pace.

On the contrary, once the mirrors system is set, it will be dead easy
to add/remove a mirror in the ring, and each node won't act as a SPOF

IMHO it's a bad idea to make this piece of our infrastructure depend
on one third party commercial entity, where we can provide a community
answer.

Now, a mirror could use Amazon, that would make more sense to me.

Regards
Tarek


>
> --
> Ian Bicking  |  http://blog.ianbicking.org
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
>



-- 
Tarek Ziadé | http://ziade.org