[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

[Tarek Ziadé]

On Tue, Jun 15, 2010 at 10:14 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Tarek Ziadé wrote:
>> On Tue, Jun 15, 2010 at 7:34 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> [..]
>>>> So I think it would be better to focus on PEP 381, and make those
>>>> existing mirrors comply with it. And maybe work on the legal issues
>>>> you've mentioned
>>>
>>> That can all happen in parallel.
>>
>> I really doubt it.
>>
>> You have come with a cloud proposal and want it to be funded by the PSF.
>>
>> Your proposal is basically a proprietary mirroring system, and it competes
>> with the mirroring protocol we wanted to build, based on the existing
>> mirrors the community has.
>
> I'm not trying to compete with your mirror PEP, just trying
> to solve a problem.

We are trying to solve the same problem, aren't we ?

That is : avoiding any downtime when PyPI is used by setuptools and
derived tools.

So if you solve this problem by implementing a cloud system backed by
a PSF funding,
and managed by the PSF, and if you claim  that there will be no more
downtime, then PEP 381
will be useless.

I am just arguing that I don't think it's the best solution, compared to what
was started e.g. a community network of mirrors.

>
>> So far I don't see any advantage in a cloud-based mirror managed by the PSF,
>> compared to a round of community mirrors.
>
> We can have it up and running in a few days and it doesn't
> require any changes to existing client tools, that's the main
> argument.

The global uptime of PyPI in this last year was probably around 99.9%,
so I don't think we are in such a rush to set up something in any case.

The problem occured in the past, and was fixed in a matter of hours.
every. time.

It's just that everytime it happens it makes us all want to improve the system.

So why don't we implement the best solution ? Maybe we could use a wiki page
and work on a synthetic overview of the pros and cons.

>
> The proposal solves a problem we have now and doesn't get in the
> way of PEP 381. Instead it buys it more time to get finalized,
> implemented and deployed on the client side.
>
> If you need funding for PEP 381, please write a proposal.

I won't.

I think we should decide here, all together, what is the best technical solution
to set up mirrors (e.g. cloud vs community)

Then, ask for its funding from the PSF.


> This would then also need to address the problem of added administration
> overhead (screening mirror server providers, getting them registered or
> removed, monitored and verified for correct operation, etc.).

This overhead is minimum compared to an in-house administration of a
full mirroring
system based on a cloud imho.

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org