[Catalog-sig] PyPI slowdowns

["Martin v. Löwis"]

> In the interest of giving people running PyPI data on problem  
> periods, PyPI struggled on several occasions over the past few days:
> 
>    Aug 25, 16:01-16:07
>    Aug 26,  9:00- 9:21
>    Aug 26, 14:56-15:00
>    Aug 27, 03:56-04:06
> 
> All of these times are UTC.
> 
> I haven't otherwise noticed problems like this for quite a while.

Thanks. I couldn't quite match all these incidences with the log files,
but apparently, what happens is this:

- some application gets overloaded (probably the Wiki, but I'm not
  certain), for some reason
- FastCGI finds that the application does not respond quickly enough,
  and kills it
- it does that a number of times, and then decides to back-off
  restarting
- as a consequence, all Apache process start blocking for that
  application. This can be seen at

http://ximinez.python.org/munin/localdomain/localhost.localdomain-apache_processes.html

  when there are 256 Apache processes.

- as a consequence, the entire web server is unaccessible, as
  the MaxClients limit is exhausted.

I don't know how detect this problem before it happens. I have
added response-time measuring to MoinMoin; if a response takes more
than 10s, it will refuse all requests with a QUERY_STRING, for
120s. As the expensive MoinMoin requests are those with query
parameters, I hope that this will cause fast processing of any
backlog that may have been built up.

Regards,
Martin