[Distutils] PyPI Status Update - PostgreSQL Tuning and CDN Shielding

[Donald Stufft]

Just a quick update on some recent PyPI changes.

We’ve attempted to tune PostgreSQL. This has resulted in a particular problem
query going from a 23s execution time to a 5s execution time. This should
reduce or remove the 503 errors that occur when the mirrors slam PyPI at
midnight UTC looking for new packages. This has reduced the bulk of the problem
with the mirrors. Assuming stability remains the same as it is now we will not
be implementing rate limiting at this time as we would greatly prefer not to
have to do that.

We’ve also activated CDN Shielding for all non POST requests. This means that
whenever you contact a node in say the singapore POP if that node does not have
the response cached instead of directly contacting the backends in Rackspace's
IAD datacenter it will instead contact the Ashburn POP. This POP will serve
a response that it has cached, or contact the backend if it does not have a
cached response. This should further speed up PyPI as it's far more likely
that the shield POP (Ashburn) will have the response cached even if the edge
location does not. Further more the connectivity between the Fastly POPs are
more likely to be lower latency then the connectivity between any random
Fastly POP and the backends.

So essentially PostgreSQL should be faster and our caching CDN should be more
efficient.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140212/c00f7e40/attachment.sig>