[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

Jesus Cea jcea at jcea.es
Tue Jun 15 19:22:22 CEST 2010

Hash: SHA1

On 15/06/10 13:49, M.-A. Lemburg wrote:
> Server side: upload cronjobs
> ----------------------------
> Since the /simple index tree is currently being created dynamically,
> we'd need to create static copies of it at regular intervals in order
> to upload the content to the S3 bucket. This can easily be done using
> tools such as wget or curl.
> Both the static copy of the /simple tree and the static files uploaded
> to /packages then need to be uploaded or updated in the S3 bucket by a
> cronjob running every 10-20 minutes.

I don't comment about the convenience to migrate or not.

But having to wait 20 minutes to deploy my just released package to my
datacenter is a bit inconvenient to me :-).

Would be nice to change PYPI code just to dump "simple" each time the
database changes. Perusing the RSS, the load should be low and actually
less demanding to CPU and database server (if you only update "simple"
with the changes, not rebuilding everything each time).

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/


More information about the Catalog-SIG mailing list