[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

[M.-A. Lemburg]

Jesus Cea wrote:
> On 15/06/10 13:49, M.-A. Lemburg wrote:
>> Server side: upload cronjobs
>> ----------------------------
> 
>> Since the /simple index tree is currently being created dynamically,
>> we'd need to create static copies of it at regular intervals in order
>> to upload the content to the S3 bucket. This can easily be done using
>> tools such as wget or curl.
> 
>> Both the static copy of the /simple tree and the static files uploaded
>> to /packages then need to be uploaded or updated in the S3 bucket by a
>> cronjob running every 10-20 minutes.
> 
> I don't comment about the convenience to migrate or not.
> 
> But having to wait 20 minutes to deploy my just released package to my
> datacenter is a bit inconvenient to me :-).
> 
> Would be nice to change PYPI code just to dump "simple" each time the
> database changes. Perusing the RSS, the load should be low and actually
> less demanding to CPU and database server (if you only update "simple"
> with the changes, not rebuilding everything each time).

I'll leave that for a version 2.0 of the cloud idea :-)

My main interest now is getting something done with only requiring
minimal changes to the PyPI software.

Note that with community servers that only mirror once a day,
you'd have to wait up to a whole day for your package updates
to become visible worldwide.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 15 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                33 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/