On Apr 26, 2011, at 4:34 PM, Ray Cote wrote:
----- Original Message -----
From: "Luke Marsden" <luke-lists@hybrid-logic.co.uk> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Tuesday, April 26, 2011 2:08:40 PM Subject: Re: [Twisted-Python] 100% CPU on high opened descriptors On Tue, 2011-04-26 at 17:42 +0200, Juan Antonio Ibañez Santorum wrote:
Hello! I have a Linux box serving my twisted app. All works ok except after some days running (where I can see my CPU going to 100%). I saw that CPU goes 100% usage when the app reaches to 1024 opened descriptors (sockets).
I assume that using a reverse proxy (Apache, nginx, etc.) in front of Twisted removes this problem? (or at least displaces it from being Twisted's problem to being the proxies problem.)
Is it recommended to always run Twisted behind a proxy?
No. It's recommended to fix bugs in Twisted that affect you :). And besides, this problem is not specific to HTTP; it affects any Twisted service that runs on a port, so there are lots of uses of Twisted where you can't run Twisted behind a proxy. Plus, this affects clients as well; if you have a twisted spidering application, your client connections could get into this same state. It's possible to handle this sort of attack without any changes to Twisted, by carefully monitoring the number of connections you have open. Of course most people just crank up their ulimits and forget about it.
We've been doing that simply to offload delivery to the final client, but I'm thinking it may have a more general need.
If you have to do it anyway then the discussion is somewhat academic, isn't it? :)