[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

[M.-A. Lemburg]

Tarek Ziadé wrote:
> On Tue, Jun 15, 2010 at 10:14 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> I'm not trying to compete with your mirror PEP, just trying
>> to solve a problem.
> 
> We are trying to solve the same problem, aren't we ?

Sure, but the intent is not to compete with the PEP. Even with
the cloud proposal implemented, we can still have a mirror
setup like the one you propose.

> That is : avoiding any downtime when PyPI is used by setuptools and
> derived tools.
>
> So if you solve this problem by implementing a cloud system backed by
> a PSF funding,
> and managed by the PSF, and if you claim  that there will be no more
> downtime, then PEP 381
> will be useless.

No, not at all. The PSF would not be the only user of the PEP
and the client tools.

If all client tools implement the things you suggested in the PEP,
we'd have a lot more possibilities.

> I am just arguing that I don't think it's the best solution, compared to what
> was started e.g. a community network of mirrors.

I've heard you, but still disagree. I think we'll just have to
leave it at that.

>>> So far I don't see any advantage in a cloud-based mirror managed by the PSF,
>>> compared to a round of community mirrors.
>>
>> We can have it up and running in a few days and it doesn't
>> require any changes to existing client tools, that's the main
>> argument.
> 
> The global uptime of PyPI in this last year was probably around 99.9%,
> so I don't think we are in such a rush to set up something in any case.
>
> The problem occured in the past, and was fixed in a matter of hours.
> every. time.
> 
> It's just that everytime it happens it makes us all want to improve the system.
> 
> So why don't we implement the best solution ? Maybe we could use a wiki page
> and work on a synthetic overview of the pros and cons.

Again: I don't want to compete against the PEP. I'm looking
for a solution that's easy to implement and doesn't get in the
way. That's all. Nothing more.

If you can come up with a solution that's ready in a month or two,
I'll happily wait.

>> The proposal solves a problem we have now and doesn't get in the
>> way of PEP 381. Instead it buys it more time to get finalized,
>> implemented and deployed on the client side.
>>
>> If you need funding for PEP 381, please write a proposal.
> 
> I won't.
> 
> I think we should decide here, all together, what is the best technical solution
> to set up mirrors (e.g. cloud vs community)
> 
> Then, ask for its funding from the PSF.
> 
> 
>> This would then also need to address the problem of added administration
>> overhead (screening mirror server providers, getting them registered or
>> removed, monitored and verified for correct operation, etc.).
> 
> This overhead is minimum compared to an in-house administration of a
> full mirroring system based on a cloud imho.

YMMV, but my experience with these systems is that they cause a lot
less overhead than anything you administer yourself.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 15 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                33 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/