[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
mal at egenix.com
Tue Jun 15 19:45:28 CEST 2010
Jesus Cea wrote:
> On 15/06/10 13:49, M.-A. Lemburg wrote:
>> Server side: upload cronjobs
>> Since the /simple index tree is currently being created dynamically,
>> we'd need to create static copies of it at regular intervals in order
>> to upload the content to the S3 bucket. This can easily be done using
>> tools such as wget or curl.
>> Both the static copy of the /simple tree and the static files uploaded
>> to /packages then need to be uploaded or updated in the S3 bucket by a
>> cronjob running every 10-20 minutes.
> I don't comment about the convenience to migrate or not.
> But having to wait 20 minutes to deploy my just released package to my
> datacenter is a bit inconvenient to me :-).
> Would be nice to change PYPI code just to dump "simple" each time the
> database changes. Perusing the RSS, the load should be low and actually
> less demanding to CPU and database server (if you only update "simple"
> with the changes, not rebuilding everything each time).
I'll leave that for a version 2.0 of the cloud idea :-)
My main interest now is getting something done with only requiring
minimal changes to the PyPI software.
Note that with community servers that only mirror once a day,
you'd have to wait up to a whole day for your package updates
to become visible worldwide.
Professional Python Services directly from the Source (#1, Jun 15 2010)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-07-19: EuroPython 2010, Birmingham, UK 33 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
More information about the Catalog-SIG