On Sun, Jun 9, 2013 at 9:33 PM, Donald Stufft
Fastly has been a dream to work with. They've been fast at fixing and diagnosing issues, have helped tune the config to get a higher hit rate, and when they heard that people were upset that download counts had to be turned off they offered the logging support to be turned on for our account.
The infrastructure is setup to receive the logs (and intact was receiving logs for a day or so) but upgrades to the VM that runs PyPI needs to occur before we can continue receiving them. The disk drive that PyPI has is to small to handle the volume of request data that is coming in from Fastly and it quickly filled up in under 24 hours. Upgrading that space requires powering off the VM so we (The Infra team) are working on doing that, ideally without downtime on PyPI.
Disk drives? It is unneeded bottleneck. What is needed is resident log cruncher that processes log entries storing intermediate results in memcache, and updates DB counts in batch. This can be even done on AppEngine. Something makes me believe that we can get some extra free quota for this purpose. =) Could you, please, share the log format+example, so that we can experiment with it? -- anatoly t.