[Catalog-sig] PyPI down again...

[M.-A. Lemburg]

Tarek Ziadé wrote:
> On Sun, Jun 13, 2010 at 3:11 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> ...
>>
>> We've had some private discussions about this, so I'm just
>> going to summarize...
>>
>> The idea here is not to override the mirror PEP ideas,
>> but to use the existing PyPI installation and put the
>> content needed for the most widely distributed package tool
>> (currently setuptools and zc.buildout) on a content
>> delivery network (CDN) in order to have it highly available
>> on a managed edge network.
> 
> I think it overlaps a bit the PEP goal, which is to set up a network of mirrors,
> and have them listed in the PyPI DNS so clients can switch from one mirror
> to another.(and even do geoloc!)
> 
> Right now we already have "unofficial mirrors" and the idea of the PEP
> would be to list them officially at PyPI and to have them collect the
> stats so we cant count download hits.

Note that the CDN does not mirror the content of PyPI, it
just takes care of delivering the requested data to the
various edge servers and caching it there for a while.

This is a different concept than that of a full mirror that
doesn't work like a cache, but instead provides a fully
functional standalone server.

I still think that the concept of being able to mirror PyPI
servers is a useful one.

>> Amazon Cloudfront is such a CDN and has Python interfaces,
>> hence the idea to use Cloudfront.
>>
>> I asked for volunteers, because I didn't know enough about
>> Amazon Cloudfront to write up a proposal and don't have
>> the cycles available to implement such a setup myself.
>>
>> In the meantime, I've done some research and now know
>> enough to write a proposal for the PSF board to consider.
>> If the board thinks it's a good idea, we'll need to
>> pursue finding volunteers to implement it.
> 
> Well maybe this is the best path to follow right now, as it will be done faster,
> without having to interact with much people to set it up, so it's a quick win.
> 
> But it will probably kill the mirroring protocol idea from the PEP in
> the process,
> which I think is superior in the long term since it provides a
> standardized ground
> for the community to set up mirrors independently from pypi.python.org.

We'll have to see.

Note that the CDN will only deal with the static data on PyPI,
not the RPC or the web GUI access.

Since static data is all that setuptools et al. currently use
for fetching the data, we'll see an improved uptime for easy_install
and esp. zc.buildout which by nature of their concepts rely on having
a high availability of the PyPI static data resources.

If, in the future, package tools start to rely on RPC for
fetching data, the situation will shift towards needing full
functional mirrors again.

OTOH, we could also provide a snapshot copy of the database
data in form of a SQLite database on the CDN for those tools
to download and use locally... there are lot's of things
package tools could do :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 14 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                34 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/