A few notes:<br><br>On Tue, Jun 29, 2010 at 9:39 AM, M.-A. Lemburg <span dir="ltr"><<a href="mailto:mal@egenix.com">mal@egenix.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
In order to maintain its credibility as software repository, to<br>
support the many different projects relying on the PyPI infrastructure<br>
and the many users who rely on the simplified installation process<br>
enabled by PyPI, the PSF needs to take action and move the essential<br>
parts of PyPI to a more robust infrastructur that provides:<br>
<br>
* scalability<br>
* 24/7 outsourced system administration management<br></blockquote><div><br>In a sense a CDN offers outsourced system administration -- if you upload content, they are responsible for making sure it gets served up. But that's all they do.<br>
<br>Other "cloud" systems only provide system administration for infrastructure issues, like a network routing issue. They do not provide anything on your machine itself. It is possible to get hosting with system administration included, Rackspace Managed Servers are an example, but these are quite expensive -- basically you are paying an overhead on hosting to have a competent sysadmin on hand. <br>
<br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Usage<br>
-----<br>
<br>
PyPI provides four different mechanisms for accessing the stored<br>
information:<br>
<br>
* a web GUI that is meant for use by humans<br>
* an RPC interface which is mostly used for uploading new<br>
content<br>
* a semi-static /simple package listing, used by setuptools<br>
* a static area /packages for package download files and<br>
documentation, used by both the web GUI and setuptools<br></blockquote><div><br>The static packages are used by the RPC (setup.py upload) and automatically linked in. There is no privileged aspect to them, Setuptools (easy_install/pip) just reads the links provided, and if they happen to point to pypi packages then that's what is fetched. I mention this because changing those URLs on the server side will be easy as a result.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
The /simple package listing is dump of all packages in PyPI using a<br>
simple HTML page with links to sub-pages for each package. These<br>
sub-pages provide links to download files and external references.<br>
<br>
External tools like easy_install only use the /simple package<br>
listing together with the hosted package download files.<br>
<br>
While the /simple package listing is currently dynamically created<br>
from the database in real-time, this is not really needed for normal<br>
operation. A static copy created every 10-20 minutes would provide the<br>
same level of service in much the same way.<br>
<br>
<br>
Moving static data to a CDN<br>
---------------------------<br>
<br>
Under the proposal the static information stored in PyPI<br>
(meta-information as well as package download files and documentation)<br>
is moved to a content delivery network (CDN).<br>
<br>
For this purpose, the /simple package listing is replaced with a<br>
static copy that is recreated every 10-20 minutes using a cronjob on<br>
the PyPI server.<br>
<br>
At the same intervals, another script will scan the package and<br>
documentation files under /packages for updates and upload any changes<br>
to the CDN for neartime availability.<br></blockquote><div><br>I disagree with this part of the proposal, because I think a 10-20 minute delay introduces the possibility of invisible errors (an infinite delay), and represents a real degradation of service as new versions of packages will not be installable until after regeneration. Also I think the RPC code (what is invoked with setup.py register/upload) can regenerate these static pages immediately.<br>
<br>Uploading to a CDN may have to be asynchronous, but to keep the data robust we should really be storing the package locally and adding a new field to point to the mirrored location (i.e., the CDN URL). When the cron job runs that field can be updated. If the CDN upload fails (which is not unlikely) then PyPI can keep using the local location. The cron job would then also be triggering another regeneration of the static file in /static, but so long as you are only regenerating on changes this isn't much overhead.<br>
<br>Also, making upload/register a synchronous operation will slow down the speed of RPC commands, but I don't think this is a problem -- I would much rather have an upload be slow to finish than fast but not know when the result will be available. I don't know what kind of latency to expect, really.<br>
<br><br>Also, I'd like to offer a counterproposal that does not use a CDN:<br><br>* Have PyPI write out static files *locally*<br>* Use rewrite rules so those files get served without touching PyPI.<br>* Move the PyPI installation to mod_wsgi (I believe it is using FCGI now?), with conservative settings for things like MaxRequests. I believe this will significantly improve the problem of PyPI taking down Apache, which means the static files will still be available even if PyPI itself is down.<br>
<br>This is largely work that would have to happen to move to a CDN, but it's simpler (given how PyPI works now) and I believe will relieve most of the problems we've seen. PyPI right now is really quite reliable, these small changes would I think be low-risk and less likely to introduce new problems while addressing what I suspect is the source of problems.<br clear="all">
<br></div></div><br>-- <br>Ian Bicking | <a href="http://blog.ianbicking.org">http://blog.ianbicking.org</a><br>